Sep 08 01:56:50.092051 ip-10-0-132-162 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 01:56:50.092061 ip-10-0-132-162 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 01:56:50.092070 ip-10-0-132-162 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 01:56:50.092365 ip-10-0-132-162 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 01:56:59.835405 ip-10-0-132-162 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 08 01:56:59.835418 ip-10-0-132-162 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3a0b7b2f6ee245df8faf2268421fdc51 -- Sep 08 01:58:38.698566 ip-10-0-132-162 systemd[1]: Starting Kubernetes Kubelet... Sep 08 01:58:39.140676 ip-10-0-132-162 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 01:58:39.140676 ip-10-0-132-162 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 01:58:39.140676 ip-10-0-132-162 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 01:58:39.140676 ip-10-0-132-162 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 01:58:39.140676 ip-10-0-132-162 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 01:58:39.143712 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.143617 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 01:58:39.147019 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.146998 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.147019 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147015 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.147019 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147020 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.147019 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147024 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147029 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147033 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147037 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147052 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147058 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147063 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147067 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147071 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147075 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147079 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147083 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147087 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147090 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147094 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147099 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147103 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147107 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147111 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.147246 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147115 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147119 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147123 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147126 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147131 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147135 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147139 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147143 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147148 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147152 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147156 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147160 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147165 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147169 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147172 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147178 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147190 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147195 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147199 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147205 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.148105 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147209 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147214 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147218 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147222 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147226 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147230 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147235 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147239 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147243 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147247 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147252 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147256 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147260 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147264 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147268 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147272 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147276 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147280 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147285 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147304 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.148977 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147308 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147312 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147317 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147321 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147325 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147332 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147337 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147344 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147349 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147364 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147369 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147373 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147377 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147381 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147385 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147389 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147399 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147405 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147410 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.149915 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147414 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147418 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147422 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147426 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147430 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147435 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147439 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147443 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147447 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147451 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147455 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147460 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147463 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147468 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147473 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147479 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147484 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.147488 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148710 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148720 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.150734 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148725 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148729 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148734 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148739 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148744 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148748 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148752 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148757 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148761 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148765 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148769 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148773 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148779 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148783 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148788 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148793 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148797 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148801 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148805 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148810 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.151624 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148814 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148818 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148823 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148827 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148831 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148835 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148839 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148844 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148848 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148851 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148855 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148859 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148865 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148870 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148875 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148879 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148884 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148889 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148893 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148896 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.152316 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148901 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148905 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148909 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148913 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148917 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148921 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148926 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148930 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148934 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148938 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148942 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148947 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148952 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148956 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148961 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148965 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148969 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148973 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148979 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148983 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.152858 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148987 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.148993 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149000 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149005 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149010 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149014 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149017 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149021 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149026 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149031 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149035 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149041 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149046 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149050 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149055 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149059 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149063 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149070 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.153393 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149075 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149080 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149085 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149089 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149093 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149097 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149102 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149107 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149111 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149115 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149119 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149123 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149128 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149132 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149136 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149140 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149144 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149148 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149153 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149770 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.154043 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149778 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149783 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149788 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149793 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149798 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149802 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149806 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149811 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149815 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149819 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149824 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149828 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149832 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149836 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149840 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149845 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149849 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149853 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149858 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149862 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.154602 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149866 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149870 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149874 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149881 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149887 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149893 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149898 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149903 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149907 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149911 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149916 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149921 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149925 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149929 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149933 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149937 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149942 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149947 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149952 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.155091 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149956 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149960 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149964 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149968 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149973 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149977 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149981 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149985 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149993 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.149997 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150001 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150005 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150009 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150014 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150018 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150022 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150026 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150030 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150034 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150038 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.155693 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150042 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150046 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150050 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150055 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150059 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150063 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150067 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150071 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150075 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150080 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150087 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150092 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150098 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150104 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150110 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150114 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150119 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150123 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150127 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.156180 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150131 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150139 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150143 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150147 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150151 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150155 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150159 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150164 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150168 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150172 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150176 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150180 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150184 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150188 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150193 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150197 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150203 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150207 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150211 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.150216 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.156691 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150335 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150388 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150404 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150411 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150419 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150425 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150431 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150439 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150444 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150449 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150455 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150460 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150466 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150473 2559 flags.go:64] FLAG: --cgroup-root="" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150478 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150484 2559 flags.go:64] FLAG: --client-ca-file="" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150489 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150494 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150500 2559 flags.go:64] FLAG: --cluster-domain="" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150505 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150510 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150515 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150520 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150526 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 01:58:39.157220 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150531 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150536 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150541 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150546 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150553 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150558 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150563 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150568 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150575 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150580 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150585 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150590 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150595 2559 flags.go:64] FLAG: --enable-server="true" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150600 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150608 2559 flags.go:64] FLAG: --event-burst="100" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150613 2559 flags.go:64] FLAG: --event-qps="50" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150618 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150623 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150628 2559 flags.go:64] FLAG: --eviction-hard="" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150634 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150639 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150646 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150651 2559 flags.go:64] FLAG: --eviction-soft="" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150655 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150660 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 01:58:39.157820 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150665 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150670 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150675 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150679 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150684 2559 flags.go:64] FLAG: --feature-gates="" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150690 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150696 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150701 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150706 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150711 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150717 2559 flags.go:64] FLAG: --help="false" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150724 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150729 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150734 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150738 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150744 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150750 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150754 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150759 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150763 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150768 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150774 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150779 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150784 2559 flags.go:64] FLAG: --kube-reserved="" Sep 08 01:58:39.158522 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150789 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150793 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150798 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150803 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150810 2559 flags.go:64] FLAG: --lock-file="" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150814 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150819 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150824 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150832 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150837 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150842 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150846 2559 flags.go:64] FLAG: --logging-format="text" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150851 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150857 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150862 2559 flags.go:64] FLAG: --manifest-url="" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150867 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150873 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150885 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150892 2559 flags.go:64] FLAG: --max-pods="110" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150899 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150904 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150908 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150913 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150918 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150923 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 01:58:39.159129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150928 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150939 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150944 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150949 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150955 2559 flags.go:64] FLAG: --pod-cidr="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150960 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150969 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150973 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150978 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150983 2559 flags.go:64] FLAG: --port="10250" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150988 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.150992 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ad7b41571f3caf82" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151000 2559 flags.go:64] FLAG: --qos-reserved="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151005 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151010 2559 flags.go:64] FLAG: --register-node="true" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151015 2559 flags.go:64] FLAG: --register-with-taints="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151022 2559 flags.go:64] FLAG: --registry-burst="10" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151026 2559 flags.go:64] FLAG: --registry-qps="5" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151031 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151035 2559 flags.go:64] FLAG: --reserved-memory="" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151041 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151046 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151051 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151056 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151060 2559 flags.go:64] FLAG: --runonce="false" Sep 08 01:58:39.159756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151065 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151073 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151078 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151082 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151087 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151092 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151097 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151102 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151107 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151111 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151116 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151121 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151126 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151131 2559 flags.go:64] FLAG: --system-cgroups="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151136 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151145 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151150 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151155 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151161 2559 flags.go:64] FLAG: --tls-min-version="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151167 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151171 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151176 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151181 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151185 2559 flags.go:64] FLAG: --v="2" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151192 2559 flags.go:64] FLAG: --version="false" Sep 08 01:58:39.160420 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151198 2559 flags.go:64] FLAG: --vmodule="" Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151204 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151209 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151404 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151412 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151417 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151422 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151426 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151433 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151439 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151446 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151451 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151456 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151461 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151465 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151470 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151474 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151478 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151482 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151486 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.161054 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151491 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151496 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151501 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151505 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151509 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151513 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151519 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151524 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151528 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151532 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151536 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151540 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151544 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151548 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151552 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151556 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151560 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151564 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151568 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151573 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.161571 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151580 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151584 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151589 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151593 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151597 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151601 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151605 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151609 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151613 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151617 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151622 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151626 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151630 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151634 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151638 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151642 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151646 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151650 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151655 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151659 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.162070 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151663 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151667 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151671 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151676 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151681 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151685 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151691 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151696 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151701 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151706 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151711 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151717 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151724 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151729 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151733 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151737 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151741 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151745 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151750 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.162573 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151754 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151758 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151762 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151766 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151770 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151774 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151778 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151782 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151787 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151791 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151796 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151800 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151804 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151808 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151812 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151816 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151820 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151824 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151828 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.163039 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151832 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151836 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151840 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.151844 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.151853 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160059 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160074 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160093 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160101 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160163 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160168 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160172 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160175 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160178 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160181 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.163551 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160184 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160187 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160190 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160193 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160196 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160198 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160202 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160205 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160208 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160211 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160213 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160216 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160219 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160222 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160225 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160228 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160230 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160233 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160236 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160238 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.163926 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160241 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160243 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160247 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160251 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160254 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160256 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160259 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160262 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160264 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160267 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160271 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160275 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160278 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160281 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160284 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160300 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160303 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160306 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160309 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.164482 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160318 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160321 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160323 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160326 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160328 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160331 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160334 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160337 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160339 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160342 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160344 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160347 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160350 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160352 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160355 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160358 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160361 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160364 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160366 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160369 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.164943 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160371 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160374 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160376 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160380 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160383 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160386 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160389 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160392 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160396 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160399 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160402 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160405 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160408 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160411 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160414 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160417 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160419 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160422 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160425 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.165464 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160428 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160430 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160433 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160435 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160438 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160441 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160444 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160447 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160450 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160452 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160455 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160458 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160460 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160463 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160465 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.165920 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160470 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160593 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160598 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160601 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160604 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160607 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160610 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160613 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160616 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160619 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160622 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160625 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160628 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160631 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160633 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160636 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160638 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160641 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160644 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160646 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 01:58:39.166303 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160649 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160651 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160655 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160659 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160662 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160665 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160668 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160671 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160674 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160677 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160680 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160682 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160685 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160688 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160690 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160693 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160696 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160699 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160702 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 01:58:39.166792 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160706 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160709 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160712 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160714 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160718 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160721 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160726 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160728 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160732 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160734 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160737 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160740 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160742 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160745 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160747 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160750 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160752 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160755 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160758 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160760 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 01:58:39.167265 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160763 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160766 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160768 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160771 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160773 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160775 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160778 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160780 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160783 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160785 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160788 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160790 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160793 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160795 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160798 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160800 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160803 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160805 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160808 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160811 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 01:58:39.167774 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160814 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160816 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160818 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160821 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160823 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160826 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160828 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160831 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160833 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160839 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160841 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160844 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160847 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160849 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160852 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160855 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160857 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160860 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160862 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160865 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 01:58:39.168271 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:39.160868 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160873 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.160971 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.163345 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.164431 2559 server.go:1019] "Starting client certificate rotation" Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.164528 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.165406 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 01:58:39.168810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.167132 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 01:58:39.173708 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.173684 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 01:58:39.185233 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.185215 2559 log.go:25] "Validated CRI v1 runtime API" Sep 08 01:58:39.189531 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.189512 2559 log.go:25] "Validated CRI v1 image API" Sep 08 01:58:39.191169 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.191155 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 01:58:39.193882 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.193865 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 01:58:39.195413 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.195387 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b2601bb-c18f-45cb-b369-1ce72f735dc3:/dev/nvme0n1p3 f4c7e5da-9165-4fb7-a3b0-a4fbfbcffeee:/dev/nvme0n1p4] Sep 08 01:58:39.195507 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.195413 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 01:58:39.200550 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.200440 2559 manager.go:216] Machine: {Timestamp:2026-09-08 01:58:39.199025688 +0000 UTC m=+0.376337751 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098161 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a114b3eaab79ef72d1f44b7282d6 SystemUUID:ec24a114-b3ea-ab79-ef72-d1f44b7282d6 BootID:3a0b7b2f-6ee2-45df-8faf-2268421fdc51 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:61:12:31:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:61:12:31:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:14:27:24:00:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 01:58:39.200550 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.200543 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 01:58:39.200701 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.200645 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 01:58:39.202748 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.202720 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 01:58:39.202910 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.202751 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 01:58:39.203638 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.203627 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 01:58:39.203693 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.203644 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 01:58:39.203693 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.203654 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 01:58:39.205306 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.205280 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 01:58:39.207204 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.207192 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 01:58:39.211867 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.211853 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 08 01:58:39.212009 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.211998 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 01:58:39.212064 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212044 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 08 01:58:39.212064 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212059 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 01:58:39.212155 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212078 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 01:58:39.212155 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212092 2559 kubelet.go:398] "Adding apiserver pod source" Sep 08 01:58:39.212155 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212105 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 01:58:39.212476 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212463 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 01:58:39.212553 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.212485 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 01:58:39.215339 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.215322 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 01:58:39.218044 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218029 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 01:58:39.218084 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218062 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 01:58:39.218217 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218205 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 01:58:39.218258 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218227 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 01:58:39.218258 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218236 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 01:58:39.218258 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218244 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 01:58:39.218258 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218252 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 01:58:39.218258 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218257 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 01:58:39.218899 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218888 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 01:58:39.218930 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218901 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 01:58:39.218930 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218908 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 01:58:39.218930 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218915 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 01:58:39.218930 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218929 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 01:58:39.219035 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.218939 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 01:58:39.219869 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.219859 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 01:58:39.219904 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.219871 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 01:58:39.223329 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.223304 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 01:58:39.223413 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.223375 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 01:58:39.223534 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.223516 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 01:58:39.224677 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.224664 2559 server.go:1268] "Started kubelet" Sep 08 01:58:39.224748 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.224709 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 01:58:39.224800 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.224738 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 01:58:39.224845 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.224815 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 01:58:39.225490 ip-10-0-132-162 systemd[1]: Started Kubernetes Kubelet. Sep 08 01:58:39.229403 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.229278 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 01:58:39.232942 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.232924 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 08 01:58:39.233031 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.232969 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 01:58:39.234935 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.233710 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-162.ec2.internal.18d335e167faf58a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-162.ec2.internal,UID:ip-10-0-132-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-162.ec2.internal,},FirstTimestamp:2026-09-08 01:58:39.224632714 +0000 UTC m=+0.401944829,LastTimestamp:2026-09-08 01:58:39.224632714 +0000 UTC m=+0.401944829,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-162.ec2.internal,}" Sep 08 01:58:39.236221 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.236200 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zpkc9" Sep 08 01:58:39.237948 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.237928 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 01:58:39.238645 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.238628 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 01:58:39.238895 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.238811 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.239511 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239493 2559 factory.go:55] Registering systemd factory Sep 08 01:58:39.239511 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239493 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 01:58:39.239651 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239515 2559 factory.go:223] Registration of the systemd container factory successfully Sep 08 01:58:39.239651 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239625 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 01:58:39.239651 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239638 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 01:58:39.239776 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239735 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 08 01:58:39.239776 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239743 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 08 01:58:39.239864 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239788 2559 factory.go:153] Registering CRI-O factory Sep 08 01:58:39.239864 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239801 2559 factory.go:223] Registration of the crio container factory successfully Sep 08 01:58:39.239864 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239849 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 01:58:39.239984 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239867 2559 factory.go:103] Registering Raw factory Sep 08 01:58:39.239984 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.239881 2559 manager.go:1196] Started watching for new ooms in manager Sep 08 01:58:39.240276 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.240263 2559 manager.go:349] Starting recovery of all containers Sep 08 01:58:39.241486 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.241465 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zpkc9" Sep 08 01:58:39.249412 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.249388 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:39.249597 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.249579 2559 manager.go:354] Recovery completed Sep 08 01:58:39.251956 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.251936 2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-162.ec2.internal\" not found" node="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.254154 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.254143 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.255952 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.255937 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.256006 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.255966 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.256006 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.255979 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.256545 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.256530 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 01:58:39.256612 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.256544 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 01:58:39.256612 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.256553 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.256612 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.256563 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 08 01:58:39.257180 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.257166 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.257252 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.257190 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.257252 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.257207 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.258641 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.258629 2559 policy_none.go:49] "None policy: Start" Sep 08 01:58:39.258693 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.258647 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 01:58:39.258693 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.258660 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 01:58:39.260700 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.260688 2559 policy_none.go:47] "Start" Sep 08 01:58:39.311365 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.311331 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 01:58:39.322764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.312452 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 01:58:39.322764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.312496 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 01:58:39.322764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.312522 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 01:58:39.322764 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.312597 2559 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 08 01:58:39.322764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.314479 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:39.339235 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.339216 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.405853 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.405691 2559 manager.go:337] "Starting Device Plugin manager" Sep 08 01:58:39.405961 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.405886 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 01:58:39.405961 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.405899 2559 server.go:87] "Starting device plugin registration server" Sep 08 01:58:39.406106 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.406096 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 01:58:39.406144 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.406108 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 01:58:39.406248 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.406232 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 01:58:39.406352 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.406340 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 01:58:39.406404 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.406353 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 01:58:39.406799 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.406782 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 01:58:39.406858 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.406834 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.413699 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.413682 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal"] Sep 08 01:58:39.413756 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.413739 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.416459 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.416440 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.416536 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.416474 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.416536 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.416488 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.418862 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.418848 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.418985 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.418973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.419026 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.418999 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.419644 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419629 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.419720 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419656 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.419720 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419671 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.419720 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419679 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.419720 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419700 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.419720 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.419713 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.421918 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.421906 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.421961 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.421930 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.422870 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.422855 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.423029 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.422887 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.423029 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.422898 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.440919 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.440903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5df1337fbaf532327312e2b3b90e53-config\") pod \"kube-apiserver-proxy-ip-10-0-132-162.ec2.internal\" (UID: \"9d5df1337fbaf532327312e2b3b90e53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.441001 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.440925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.441001 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.440943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.449818 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.449796 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-162.ec2.internal\" not found" node="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.453530 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.453516 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-162.ec2.internal\" not found" node="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.506475 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.506458 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 01:58:39.507275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.507261 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientMemory" Sep 08 01:58:39.507350 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.507305 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 01:58:39.507350 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.507320 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeHasSufficientPID" Sep 08 01:58:39.507350 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.507346 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.515667 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.515653 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.515715 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.515674 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-162.ec2.internal\": node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.538669 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.538647 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.541836 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.541889 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.541889 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5df1337fbaf532327312e2b3b90e53-config\") pod \"kube-apiserver-proxy-ip-10-0-132-162.ec2.internal\" (UID: \"9d5df1337fbaf532327312e2b3b90e53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.541959 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d5df1337fbaf532327312e2b3b90e53-config\") pod \"kube-apiserver-proxy-ip-10-0-132-162.ec2.internal\" (UID: \"9d5df1337fbaf532327312e2b3b90e53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.541959 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.541959 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.541926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5653829a7db61e4040e399d603bee343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal\" (UID: \"5653829a7db61e4040e399d603bee343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.639051 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.639020 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.739734 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.739660 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.753953 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.753934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.757777 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:39.757750 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:39.840970 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.840935 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:39.941540 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:39.941493 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.042631 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:40.042560 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.144215 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:40.144180 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.164977 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.164950 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 01:58:40.165105 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.165089 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 01:58:40.165152 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.165125 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 01:58:40.226104 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.226083 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:40.244741 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:40.244717 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.244828 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.244736 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 01:53:39 +0000 UTC" deadline="2028-02-14 10:25:29.895186427 +0000 UTC" Sep 08 01:58:40.244828 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.244765 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12584h26m49.650424876s" Sep 08 01:58:40.344946 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:40.344882 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.446102 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:40.446083 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-162.ec2.internal\" not found" Sep 08 01:58:40.457394 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.457358 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:40.539595 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.539312 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" Sep 08 01:58:40.547652 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.547633 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 01:58:40.548885 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.548868 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" Sep 08 01:58:40.559776 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.559758 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 01:58:40.632719 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:40.632676 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5653829a7db61e4040e399d603bee343.slice/crio-1018bc36ab6d8b51b073244b03ed74552e36afe8ef2387788777dd47d877ebe6 WatchSource:0}: Error finding container 1018bc36ab6d8b51b073244b03ed74552e36afe8ef2387788777dd47d877ebe6: Status 404 returned error can't find the container with id 1018bc36ab6d8b51b073244b03ed74552e36afe8ef2387788777dd47d877ebe6 Sep 08 01:58:40.638071 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:40.638052 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5df1337fbaf532327312e2b3b90e53.slice/crio-de64c120dfaf789c3cb3145f42ffe70a8f522fc1f3dec5b31adbd7c9f159e416 WatchSource:0}: Error finding container de64c120dfaf789c3cb3145f42ffe70a8f522fc1f3dec5b31adbd7c9f159e416: Status 404 returned error can't find the container with id de64c120dfaf789c3cb3145f42ffe70a8f522fc1f3dec5b31adbd7c9f159e416 Sep 08 01:58:40.640421 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:40.640406 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 01:58:41.215167 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.215133 2559 apiserver.go:52] "Watching apiserver" Sep 08 01:58:41.224940 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.224914 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 01:58:41.227883 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.227860 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal","openshift-dns/node-resolver-5496l","openshift-multus/multus-6x78z","openshift-multus/network-metrics-daemon-7c8t2","openshift-network-diagnostics/network-check-target-9p5g4","kube-system/konnectivity-agent-qhfwz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77","openshift-cluster-node-tuning-operator/tuned-64wjx","openshift-image-registry/node-ca-zmcsm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-9jpxh","openshift-network-operator/iptables-alerter-j965l","openshift-ovn-kubernetes/ovnkube-node-5hj6f","kube-system/global-pull-secret-syncer-wpgqg"] Sep 08 01:58:41.231603 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.231584 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.233978 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.233958 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.234104 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.234049 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 01:58:41.234177 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.234047 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 01:58:41.234240 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.234052 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2vtb\"" Sep 08 01:58:41.236015 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.235998 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 01:58:41.236295 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236277 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.236651 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 01:58:41.236823 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236803 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wjlgx\"" Sep 08 01:58:41.236889 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236822 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.236889 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236851 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.237002 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.236896 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 01:58:41.237045 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.237000 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 01:58:41.238212 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.238196 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 01:58:41.238494 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.238476 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nlrpv\"" Sep 08 01:58:41.238807 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.238781 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 01:58:41.238890 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.238858 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.238890 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.238878 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.239129 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.239108 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.240914 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.240899 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kz4hl\"" Sep 08 01:58:41.241013 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.240914 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 01:58:41.241013 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.240917 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.241013 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.240942 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.241318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.241291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.243121 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.243102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 01:58:41.243215 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.243190 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.243282 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.243218 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.243566 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.243547 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx8f9\"" Sep 08 01:58:41.243696 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.243680 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:41.243752 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.243735 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:41.246045 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.246030 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.247513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.247489 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 01:58:41.248193 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.248174 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.248278 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.248215 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.248336 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.248307 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwsdw\"" Sep 08 01:58:41.249039 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.249022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.249108 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.249093 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:41.250018 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.249997 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69d8342c-24ae-4b85-87d9-bc9ce116e096-konnectivity-ca\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.250094 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-systemd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250094 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-var-lib-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250094 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250094 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-registration-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-socket-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250225 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-script-lib\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250272 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/128d5c0b-cee0-460b-a941-64eb1bd1c95a-serviceca\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-netd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250346 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-env-overrides\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfsch\" (UniqueName: \"kubernetes.io/projected/ec5768d3-add2-4b17-9b63-774b9810ffd0-kube-api-access-nfsch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-sys-fs\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-kubelet\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-netns\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-ovn\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-log-socket\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d5c0b-cee0-460b-a941-64eb1bd1c95a-host\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.250636 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69d8342c-24ae-4b85-87d9-bc9ce116e096-agent-certs\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-slash\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-node-log\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250691 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72439b0a-1314-42c1-b32a-ecdea6839bb5-iptables-alerter-script\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72439b0a-1314-42c1-b32a-ecdea6839bb5-host-slash\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9pg\" (UniqueName: \"kubernetes.io/projected/128d5c0b-cee0-460b-a941-64eb1bd1c95a-kube-api-access-7h9pg\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-etc-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-config\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-systemd-units\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-bin\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglk2\" (UniqueName: \"kubernetes.io/projected/72439b0a-1314-42c1-b32a-ecdea6839bb5-kube-api-access-mglk2\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-device-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.251079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.250984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqvz\" (UniqueName: \"kubernetes.io/projected/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kube-api-access-glqvz\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.251688 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.251672 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.253906 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.253890 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 01:58:41.254007 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.253938 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.254187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.254168 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 01:58:41.254338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.254285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.254428 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.254412 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zs9wd\"" Sep 08 01:58:41.255580 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.254533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 01:58:41.255580 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.255045 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.258249 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.257185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-djzfq\"" Sep 08 01:58:41.258249 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.257251 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 01:58:41.259797 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.259774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.261955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.261930 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 01:58:41.262124 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.261927 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 01:58:41.262266 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.261945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5xfgw\"" Sep 08 01:58:41.262552 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.262538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.262698 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.262673 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:41.276802 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.276779 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w28sq" Sep 08 01:58:41.284724 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.284701 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w28sq" Sep 08 01:58:41.320334 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.320268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" event={"ID":"9d5df1337fbaf532327312e2b3b90e53","Type":"ContainerStarted","Data":"de64c120dfaf789c3cb3145f42ffe70a8f522fc1f3dec5b31adbd7c9f159e416"} Sep 08 01:58:41.322221 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.322190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" event={"ID":"5653829a7db61e4040e399d603bee343","Type":"ContainerStarted","Data":"1018bc36ab6d8b51b073244b03ed74552e36afe8ef2387788777dd47d877ebe6"} Sep 08 01:58:41.340931 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.340907 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 01:58:41.351569 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.351681 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-registration-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.351681 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-conf\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351681 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.351681 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-script-lib\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvn9l\" (UniqueName: \"kubernetes.io/projected/082e2a56-3930-4c0a-99b7-910f08509976-kube-api-access-kvn9l\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-dbus\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysconfig\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-kubelet-config\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-systemd\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-var-lib-kubelet\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-tuned\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.351873 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-tmp\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-system-cni-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-registration-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.351971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-kubelet\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-netns\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-ovn\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-kubelet\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d5c0b-cee0-460b-a941-64eb1bd1c95a-host\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-ovn\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d5c0b-cee0-460b-a941-64eb1bd1c95a-host\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-netns\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tljs\" (UniqueName: \"kubernetes.io/projected/1f1566da-bfb8-430f-9d27-4f096b4ec427-kube-api-access-7tljs\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.352318 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69d8342c-24ae-4b85-87d9-bc9ce116e096-agent-certs\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-node-log\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9pg\" (UniqueName: \"kubernetes.io/projected/128d5c0b-cee0-460b-a941-64eb1bd1c95a-kube-api-access-7h9pg\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-system-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-kubernetes\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-config\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-netns\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-systemd-units\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-node-log\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglk2\" (UniqueName: \"kubernetes.io/projected/72439b0a-1314-42c1-b32a-ecdea6839bb5-kube-api-access-mglk2\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-device-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-hostroot\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-daemon-config\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xw6d\" (UniqueName: \"kubernetes.io/projected/81fbebbd-3af9-454e-83cd-a0578da0d9ab-kube-api-access-8xw6d\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-systemd-units\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352918 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.353087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352926 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-device-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.352972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69d8342c-24ae-4b85-87d9-bc9ce116e096-konnectivity-ca\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-systemd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-var-lib-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cni-binary-copy\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-conf-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353164 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-script-lib\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353172 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovnkube-config\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-modprobe-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-run\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-socket-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-var-lib-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-sys\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb55x\" (UniqueName: \"kubernetes.io/projected/a18d844d-f8e0-40fe-a647-5fafbe63519c-kube-api-access-jb55x\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.353908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-run-systemd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-socket-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-cnibin\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/128d5c0b-cee0-460b-a941-64eb1bd1c95a-serviceca\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-etc-kubernetes\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69d8342c-24ae-4b85-87d9-bc9ce116e096-konnectivity-ca\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e2a56-3930-4c0a-99b7-910f08509976-tmp-dir\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-netd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-env-overrides\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfsch\" (UniqueName: \"kubernetes.io/projected/ec5768d3-add2-4b17-9b63-774b9810ffd0-kube-api-access-nfsch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-netd\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.353985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-sys-fs\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.354642 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/128d5c0b-cee0-460b-a941-64eb1bd1c95a-serviceca\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-log-socket\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca4f623-fdca-483b-b1e1-e0719251c2ab-sys-fs\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e2a56-3930-4c0a-99b7-910f08509976-hosts-file\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-log-socket\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-multus-certs\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354290 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-slash\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-slash\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72439b0a-1314-42c1-b32a-ecdea6839bb5-iptables-alerter-script\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72439b0a-1314-42c1-b32a-ecdea6839bb5-host-slash\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354431 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cnibin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-k8s-cni-cncf-io\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-multus\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec5768d3-add2-4b17-9b63-774b9810ffd0-env-overrides\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvm9\" (UniqueName: \"kubernetes.io/projected/bf2486a7-aa44-4bbd-b047-262593d0ad76-kube-api-access-cgvm9\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-etc-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.355513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-socket-dir-parent\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-bin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354637 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-bin\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354691 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqvz\" (UniqueName: \"kubernetes.io/projected/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kube-api-access-glqvz\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-kubelet\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-lib-modules\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-host\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72439b0a-1314-42c1-b32a-ecdea6839bb5-iptables-alerter-script\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-host-cni-bin\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.354994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72439b0a-1314-42c1-b32a-ecdea6839bb5-host-slash\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.356338 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.355009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec5768d3-add2-4b17-9b63-774b9810ffd0-etc-openvswitch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.357736 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.357712 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5768d3-add2-4b17-9b63-774b9810ffd0-ovn-node-metrics-cert\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.358306 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.358286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69d8342c-24ae-4b85-87d9-bc9ce116e096-agent-certs\") pod \"konnectivity-agent-qhfwz\" (UID: \"69d8342c-24ae-4b85-87d9-bc9ce116e096\") " pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.361510 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.361480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9pg\" (UniqueName: \"kubernetes.io/projected/128d5c0b-cee0-460b-a941-64eb1bd1c95a-kube-api-access-7h9pg\") pod \"node-ca-zmcsm\" (UID: \"128d5c0b-cee0-460b-a941-64eb1bd1c95a\") " pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.361612 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.361563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglk2\" (UniqueName: \"kubernetes.io/projected/72439b0a-1314-42c1-b32a-ecdea6839bb5-kube-api-access-mglk2\") pod \"iptables-alerter-j965l\" (UID: \"72439b0a-1314-42c1-b32a-ecdea6839bb5\") " pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.366968 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.366948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfsch\" (UniqueName: \"kubernetes.io/projected/ec5768d3-add2-4b17-9b63-774b9810ffd0-kube-api-access-nfsch\") pod \"ovnkube-node-5hj6f\" (UID: \"ec5768d3-add2-4b17-9b63-774b9810ffd0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.368799 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.368613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqvz\" (UniqueName: \"kubernetes.io/projected/2ca4f623-fdca-483b-b1e1-e0719251c2ab-kube-api-access-glqvz\") pod \"aws-ebs-csi-driver-node-4rh77\" (UID: \"2ca4f623-fdca-483b-b1e1-e0719251c2ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.402059 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.402029 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:41.456045 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-etc-kubernetes\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e2a56-3930-4c0a-99b7-910f08509976-tmp-dir\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e2a56-3930-4c0a-99b7-910f08509976-hosts-file\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.456229 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-multus-certs\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cnibin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456274 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-k8s-cni-cncf-io\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-multus\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456315 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvm9\" (UniqueName: \"kubernetes.io/projected/bf2486a7-aa44-4bbd-b047-262593d0ad76-kube-api-access-cgvm9\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-socket-dir-parent\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-bin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456414 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-kubelet\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-lib-modules\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-host\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-conf\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvn9l\" (UniqueName: \"kubernetes.io/projected/082e2a56-3930-4c0a-99b7-910f08509976-kube-api-access-kvn9l\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-dbus\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.456592 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysconfig\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-kubelet-config\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-systemd\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-var-lib-kubelet\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-tuned\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-tmp\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-system-cni-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tljs\" (UniqueName: \"kubernetes.io/projected/1f1566da-bfb8-430f-9d27-4f096b4ec427-kube-api-access-7tljs\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-system-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-kubernetes\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-netns\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-hostroot\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.456977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-daemon-config\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xw6d\" (UniqueName: \"kubernetes.io/projected/81fbebbd-3af9-454e-83cd-a0578da0d9ab-kube-api-access-8xw6d\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.457348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cni-binary-copy\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-conf-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-modprobe-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-run\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-sys\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb55x\" (UniqueName: \"kubernetes.io/projected/a18d844d-f8e0-40fe-a647-5fafbe63519c-kube-api-access-jb55x\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-cnibin\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-cnibin\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.458275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.457972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.458755 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.458755 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-systemd\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458755 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458609 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-var-lib-kubelet\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.458755 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e2a56-3930-4c0a-99b7-910f08509976-tmp-dir\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.458958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-etc-kubernetes\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.458958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458869 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.458958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e2a56-3930-4c0a-99b7-910f08509976-hosts-file\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.458958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-multus-certs\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.459430 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f1566da-bfb8-430f-9d27-4f096b4ec427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.459516 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cnibin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.459587 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-k8s-cni-cncf-io\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.459682 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-multus\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.459933 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-socket-dir-parent\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.459999 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.459957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-cni-bin\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.460059 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.460027 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:41.460114 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.460081 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:41.960060249 +0000 UTC m=+3.137372300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:41.460243 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-var-lib-kubelet\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.460382 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-lib-modules\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.460453 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-host\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.460546 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-conf\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.460769 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-dbus\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.460846 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysconfig\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.460976 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.460963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-sysctl-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.461224 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-system-cni-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.461411 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-tuned\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.461487 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-modprobe-d\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.461594 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.461567 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:41.461654 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.461626 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:41.961605544 +0000 UTC m=+3.138917594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:41.461654 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-system-cni-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.461766 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-etc-kubernetes\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.461766 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-host-run-netns\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.461766 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461734 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-hostroot\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.461942 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.461791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f1566da-bfb8-430f-9d27-4f096b4ec427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.462396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.462093 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-conf-dir\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.462396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.462095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-run\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.462396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.462151 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18d844d-f8e0-40fe-a647-5fafbe63519c-sys\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.462396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.458464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-kubelet-config\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.462396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.462244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-multus-daemon-config\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.462801 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.462777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81fbebbd-3af9-454e-83cd-a0578da0d9ab-cni-binary-copy\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.463227 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.463208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18d844d-f8e0-40fe-a647-5fafbe63519c-tmp\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.466733 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.466687 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:41.466733 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.466713 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:41.466733 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.466727 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:41.466876 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.466779 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:58:41.966762111 +0000 UTC m=+3.144074181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:41.472116 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.471930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvn9l\" (UniqueName: \"kubernetes.io/projected/082e2a56-3930-4c0a-99b7-910f08509976-kube-api-access-kvn9l\") pod \"node-resolver-5496l\" (UID: \"082e2a56-3930-4c0a-99b7-910f08509976\") " pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.474407 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.474182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvm9\" (UniqueName: \"kubernetes.io/projected/bf2486a7-aa44-4bbd-b047-262593d0ad76-kube-api-access-cgvm9\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:41.474757 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.474714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb55x\" (UniqueName: \"kubernetes.io/projected/a18d844d-f8e0-40fe-a647-5fafbe63519c-kube-api-access-jb55x\") pod \"tuned-64wjx\" (UID: \"a18d844d-f8e0-40fe-a647-5fafbe63519c\") " pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.476040 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.476015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tljs\" (UniqueName: \"kubernetes.io/projected/1f1566da-bfb8-430f-9d27-4f096b4ec427-kube-api-access-7tljs\") pod \"multus-additional-cni-plugins-9jpxh\" (UID: \"1f1566da-bfb8-430f-9d27-4f096b4ec427\") " pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.476219 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.476184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xw6d\" (UniqueName: \"kubernetes.io/projected/81fbebbd-3af9-454e-83cd-a0578da0d9ab-kube-api-access-8xw6d\") pod \"multus-6x78z\" (UID: \"81fbebbd-3af9-454e-83cd-a0578da0d9ab\") " pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.542506 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.542471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:58:41.549243 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.549219 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:58:41.559143 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.559115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j965l" Sep 08 01:58:41.559441 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.559413 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d8342c_24ae_4b85_87d9_bc9ce116e096.slice/crio-a0c8cedf507b63321f45a4bb993951be96827b37c912d598f80161ca717a8053 WatchSource:0}: Error finding container a0c8cedf507b63321f45a4bb993951be96827b37c912d598f80161ca717a8053: Status 404 returned error can't find the container with id a0c8cedf507b63321f45a4bb993951be96827b37c912d598f80161ca717a8053 Sep 08 01:58:41.562274 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.562049 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5768d3_add2_4b17_9b63_774b9810ffd0.slice/crio-bfcb2cabef411b9c1eb5c3002e0fb86deaa3c099135ce911eb7667df0369f567 WatchSource:0}: Error finding container bfcb2cabef411b9c1eb5c3002e0fb86deaa3c099135ce911eb7667df0369f567: Status 404 returned error can't find the container with id bfcb2cabef411b9c1eb5c3002e0fb86deaa3c099135ce911eb7667df0369f567 Sep 08 01:58:41.565955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.565759 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmcsm" Sep 08 01:58:41.574227 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.574196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" Sep 08 01:58:41.576019 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.575993 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72439b0a_1314_42c1_b32a_ecdea6839bb5.slice/crio-2aada498a598a91a3f669d979ec66a9dd8b543d142b82d813b61abfd20438225 WatchSource:0}: Error finding container 2aada498a598a91a3f669d979ec66a9dd8b543d142b82d813b61abfd20438225: Status 404 returned error can't find the container with id 2aada498a598a91a3f669d979ec66a9dd8b543d142b82d813b61abfd20438225 Sep 08 01:58:41.583171 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.583145 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128d5c0b_cee0_460b_a941_64eb1bd1c95a.slice/crio-0ee3b534883bce274c1add85d01b6bba044e4d2c9732c6ede6e54c632c73e39b WatchSource:0}: Error finding container 0ee3b534883bce274c1add85d01b6bba044e4d2c9732c6ede6e54c632c73e39b: Status 404 returned error can't find the container with id 0ee3b534883bce274c1add85d01b6bba044e4d2c9732c6ede6e54c632c73e39b Sep 08 01:58:41.583271 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.583257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-64wjx" Sep 08 01:58:41.590612 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.590589 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" Sep 08 01:58:41.593863 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.593827 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca4f623_fdca_483b_b1e1_e0719251c2ab.slice/crio-be5d8592f03435b8da3abf6a9844a4b8a4d05e8c7f874732fd7e2edd2d43f5bc WatchSource:0}: Error finding container be5d8592f03435b8da3abf6a9844a4b8a4d05e8c7f874732fd7e2edd2d43f5bc: Status 404 returned error can't find the container with id be5d8592f03435b8da3abf6a9844a4b8a4d05e8c7f874732fd7e2edd2d43f5bc Sep 08 01:58:41.598972 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.598594 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6x78z" Sep 08 01:58:41.599168 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.599143 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18d844d_f8e0_40fe_a647_5fafbe63519c.slice/crio-9576eb4ce6d499a1fd9d41bc0d6d23f69634fc72a8a44da9c6a1aa8f61d91d5b WatchSource:0}: Error finding container 9576eb4ce6d499a1fd9d41bc0d6d23f69634fc72a8a44da9c6a1aa8f61d91d5b: Status 404 returned error can't find the container with id 9576eb4ce6d499a1fd9d41bc0d6d23f69634fc72a8a44da9c6a1aa8f61d91d5b Sep 08 01:58:41.604633 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.604608 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 01:58:41.606259 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.606240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5496l" Sep 08 01:58:41.619498 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:58:41.619473 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082e2a56_3930_4c0a_99b7_910f08509976.slice/crio-ab6335d1bf55a36b96634f197c32f13a0a593a5283fe85584bb3f735c3830d11 WatchSource:0}: Error finding container ab6335d1bf55a36b96634f197c32f13a0a593a5283fe85584bb3f735c3830d11: Status 404 returned error can't find the container with id ab6335d1bf55a36b96634f197c32f13a0a593a5283fe85584bb3f735c3830d11 Sep 08 01:58:41.961351 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:41.961243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:41.961517 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.961419 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:41.961517 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:41.961494 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:42.961475782 +0000 UTC m=+4.138787850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:42.061963 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.061920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:42.062153 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.061976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:42.062153 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062131 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:42.062153 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062145 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:42.062345 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062162 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:42.062345 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062180 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:42.062345 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062218 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:43.062197993 +0000 UTC m=+4.239510064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:42.062345 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.062240 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:58:43.062229317 +0000 UTC m=+4.239541376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:42.290239 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.290124 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 01:53:41 +0000 UTC" deadline="2028-04-03 14:32:43.378821717 +0000 UTC" Sep 08 01:58:42.290239 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.290171 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13764h34m1.088657105s" Sep 08 01:58:42.313488 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.313446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:42.313676 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.313592 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:42.326131 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.326089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5496l" event={"ID":"082e2a56-3930-4c0a-99b7-910f08509976","Type":"ContainerStarted","Data":"ab6335d1bf55a36b96634f197c32f13a0a593a5283fe85584bb3f735c3830d11"} Sep 08 01:58:42.327619 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.327591 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6x78z" event={"ID":"81fbebbd-3af9-454e-83cd-a0578da0d9ab","Type":"ContainerStarted","Data":"f1d9aeef3dc64e10b0056e2b28f9f124f71bd3e9009b16db856d89217ad8896a"} Sep 08 01:58:42.329345 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.329293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerStarted","Data":"f7303b71d22dd331d6af3dda0a33a5248a68ecaa011f6a7899dc605d69891ed5"} Sep 08 01:58:42.330763 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.330736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-64wjx" event={"ID":"a18d844d-f8e0-40fe-a647-5fafbe63519c","Type":"ContainerStarted","Data":"9576eb4ce6d499a1fd9d41bc0d6d23f69634fc72a8a44da9c6a1aa8f61d91d5b"} Sep 08 01:58:42.332317 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.332274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" event={"ID":"2ca4f623-fdca-483b-b1e1-e0719251c2ab","Type":"ContainerStarted","Data":"be5d8592f03435b8da3abf6a9844a4b8a4d05e8c7f874732fd7e2edd2d43f5bc"} Sep 08 01:58:42.333787 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.333760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmcsm" event={"ID":"128d5c0b-cee0-460b-a941-64eb1bd1c95a","Type":"ContainerStarted","Data":"0ee3b534883bce274c1add85d01b6bba044e4d2c9732c6ede6e54c632c73e39b"} Sep 08 01:58:42.335268 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.335241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j965l" event={"ID":"72439b0a-1314-42c1-b32a-ecdea6839bb5","Type":"ContainerStarted","Data":"2aada498a598a91a3f669d979ec66a9dd8b543d142b82d813b61abfd20438225"} Sep 08 01:58:42.336708 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.336681 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"bfcb2cabef411b9c1eb5c3002e0fb86deaa3c099135ce911eb7667df0369f567"} Sep 08 01:58:42.338276 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.338247 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhfwz" event={"ID":"69d8342c-24ae-4b85-87d9-bc9ce116e096","Type":"ContainerStarted","Data":"a0c8cedf507b63321f45a4bb993951be96827b37c912d598f80161ca717a8053"} Sep 08 01:58:42.970192 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:42.970145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:42.971010 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.970382 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:42.971010 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:42.970456 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:44.970433976 +0000 UTC m=+6.147746031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.070629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.070692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.070831 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.070900 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:45.070878212 +0000 UTC m=+6.248190269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.071317 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.071337 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.071351 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:43.071513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.071412 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:58:45.07139688 +0000 UTC m=+6.248708998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:43.290698 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.290581 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 01:53:41 +0000 UTC" deadline="2028-03-21 22:57:19.002445214 +0000 UTC" Sep 08 01:58:43.290698 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.290648 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13460h58m35.711802356s" Sep 08 01:58:43.314116 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.314079 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:43.314293 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.314220 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:43.314360 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:43.314083 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:43.314513 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:43.314474 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:44.313417 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:44.313115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:44.314053 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:44.313478 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:44.373308 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:44.373245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" event={"ID":"9d5df1337fbaf532327312e2b3b90e53","Type":"ContainerStarted","Data":"a7e004da18c222a9343d5236d2378c8eaffec38350fd5c633d8a713fbbc3ee78"} Sep 08 01:58:44.383836 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:44.383802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" event={"ID":"5653829a7db61e4040e399d603bee343","Type":"ContainerStarted","Data":"3b95874d8e239ce2620ed3b6627dc27e35fe878ac563101f61fcb4df8ce705b8"} Sep 08 01:58:44.401104 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:44.400501 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-162.ec2.internal" podStartSLOduration=4.40047943 podStartE2EDuration="4.40047943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:58:44.388096201 +0000 UTC m=+5.565408275" watchObservedRunningTime="2026-09-08 01:58:44.40047943 +0000 UTC m=+5.577791500" Sep 08 01:58:44.988346 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:44.988277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:44.988533 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:44.988422 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:44.988533 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:44.988499 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:48.988479632 +0000 UTC m=+10.165791694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:45.089273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:45.089340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089430 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089443 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089455 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089469 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089506 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:49.089488059 +0000 UTC m=+10.266800124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:45.089871 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.089525 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:58:49.089514843 +0000 UTC m=+10.266826897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:45.313507 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:45.313390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:45.313925 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.313536 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:45.313925 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:45.313860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:45.314049 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:45.313953 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:46.313811 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:46.313777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:46.314266 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:46.313916 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:46.391520 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:46.391486 2559 generic.go:356] "Generic (PLEG): container finished" podID="5653829a7db61e4040e399d603bee343" containerID="3b95874d8e239ce2620ed3b6627dc27e35fe878ac563101f61fcb4df8ce705b8" exitCode=0 Sep 08 01:58:46.391669 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:46.391540 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" event={"ID":"5653829a7db61e4040e399d603bee343","Type":"ContainerDied","Data":"3b95874d8e239ce2620ed3b6627dc27e35fe878ac563101f61fcb4df8ce705b8"} Sep 08 01:58:47.313668 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:47.313579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:47.313668 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:47.313596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:47.313880 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:47.313709 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:47.313880 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:47.313831 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:48.313902 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:48.313363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:48.313902 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:48.313512 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:49.024511 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:49.024473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:49.024675 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.024614 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:49.024744 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.024685 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:57.024670078 +0000 UTC m=+18.201982142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:49.124777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:49.124839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.124914 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.124938 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.124951 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.124967 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.125003 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:58:57.124986639 +0000 UTC m=+18.302298702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:49.125009 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.125021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:57.125012189 +0000 UTC m=+18.302324244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:49.314281 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:49.314196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:49.314721 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.314313 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:49.314721 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:49.314374 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:49.314721 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:49.314472 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:50.313903 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:50.313470 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:50.313903 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:50.313601 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:51.312915 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:51.312888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:51.313491 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:51.312925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:51.313491 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:51.312985 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:51.313491 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:51.313059 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:52.313398 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:52.313362 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:52.313807 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:52.313480 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:53.315881 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:53.315839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:53.316265 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:53.315988 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:53.316265 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:53.316041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:53.316265 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:53.316127 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:54.313225 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:54.313194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:54.313401 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:54.313327 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:55.313423 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:55.313389 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:55.313870 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:55.313544 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:55.313870 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:55.313589 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:55.313870 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:55.313701 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:56.312983 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:56.312952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:56.313100 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:56.313043 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:57.082852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:57.082808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:57.083551 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.082931 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:57.083551 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.082987 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:13.082971265 +0000 UTC m=+34.260283328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:58:57.184106 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:57.184071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:57.184106 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:57.184113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184205 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184229 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184257 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184264 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:13.184245794 +0000 UTC m=+34.361557847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184269 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:57.184398 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.184334 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:59:13.184318992 +0000 UTC m=+34.361631056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:58:57.316532 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:57.316506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:57.316532 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:57.316541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:57.316761 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.316626 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:58:57.316761 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:57.316709 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:58.313619 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:58.313586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:58:58.314091 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:58.313706 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:58:59.313943 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:59.313896 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:58:59.314405 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:58:59.313985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:58:59.314405 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:59.314008 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:58:59.314405 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:58:59.314108 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:00.313883 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.313704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:00.314000 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:00.313981 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:00.422001 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.421968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmcsm" event={"ID":"128d5c0b-cee0-460b-a941-64eb1bd1c95a","Type":"ContainerStarted","Data":"ae3c93481221e2bc1aa1c3b0156eab94ac2f14e3481be3c8bb26450d98dc8b80"} Sep 08 01:59:00.424071 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.424039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" event={"ID":"5653829a7db61e4040e399d603bee343","Type":"ContainerStarted","Data":"6e69baca7459a0ad2cc7fdc7f2972571a39d0e5c5bb47e86c2be4834a1fd7697"} Sep 08 01:59:00.425475 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.425433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhfwz" event={"ID":"69d8342c-24ae-4b85-87d9-bc9ce116e096","Type":"ContainerStarted","Data":"e5e3c3bb644aab150d24fb4c9023809492f5760bf486d82e9b2a98fd38ce0645"} Sep 08 01:59:00.427004 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.426977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5496l" event={"ID":"082e2a56-3930-4c0a-99b7-910f08509976","Type":"ContainerStarted","Data":"f99c94bf74ae10c80b31c07c462846dc7e6303971cbcd5ecfe2bf70bf41e9bd9"} Sep 08 01:59:00.428718 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.428683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6x78z" event={"ID":"81fbebbd-3af9-454e-83cd-a0578da0d9ab","Type":"ContainerStarted","Data":"d238032e984446c37466cb953be3b29ec9608a9758bee222888f8a53252d2477"} Sep 08 01:59:00.430357 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.430328 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="cecc16c75f6af17e941909785ae72aa8b12e6ae6070e98592d18d6bd90b3e3a5" exitCode=0 Sep 08 01:59:00.430446 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.430403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"cecc16c75f6af17e941909785ae72aa8b12e6ae6070e98592d18d6bd90b3e3a5"} Sep 08 01:59:00.432282 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.432254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-64wjx" event={"ID":"a18d844d-f8e0-40fe-a647-5fafbe63519c","Type":"ContainerStarted","Data":"9500d27c55ee6d0a598f625b5fa0554c9db80e54a10a51afb21977cb6111fe38"} Sep 08 01:59:00.436774 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.436402 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zmcsm" podStartSLOduration=6.77594003 podStartE2EDuration="21.436334383s" totalImagesPullingTime="14.660394353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.435862966 +0000 UTC m=+21.613175036" watchObservedRunningTime="2026-09-08 01:59:00.436334383 +0000 UTC m=+21.613646455" Sep 08 01:59:00.437554 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.437533 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" event={"ID":"2ca4f623-fdca-483b-b1e1-e0719251c2ab","Type":"ContainerStarted","Data":"94b465a0c35d6d10c135cf13e09dc47ff4a1479b9060b84a0ab190333b2be780"} Sep 08 01:59:00.448513 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.448474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-162.ec2.internal" podStartSLOduration=20.44845532 podStartE2EDuration="20.44845532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.448361592 +0000 UTC m=+21.625673664" watchObservedRunningTime="2026-09-08 01:59:00.44845532 +0000 UTC m=+21.625767393" Sep 08 01:59:00.479389 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.479335 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhfwz" podStartSLOduration=3.054913313 podStartE2EDuration="21.479320714s" totalImagesPullingTime="18.424407401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.479153129 +0000 UTC m=+21.656465204" watchObservedRunningTime="2026-09-08 01:59:00.479320714 +0000 UTC m=+21.656632783" Sep 08 01:59:00.494081 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.494047 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-64wjx" podStartSLOduration=2.094094359 podStartE2EDuration="20.494031281s" totalImagesPullingTime="18.399936922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.49388533 +0000 UTC m=+21.671197403" watchObservedRunningTime="2026-09-08 01:59:00.494031281 +0000 UTC m=+21.671343353" Sep 08 01:59:00.533881 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.533722 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5496l" podStartSLOduration=2.169298792 podStartE2EDuration="20.533706954s" totalImagesPullingTime="18.364408162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.533113429 +0000 UTC m=+21.710425502" watchObservedRunningTime="2026-09-08 01:59:00.533706954 +0000 UTC m=+21.711019026" Sep 08 01:59:00.534008 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:00.533932 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6x78z" podStartSLOduration=2.122640729 podStartE2EDuration="20.533922623s" totalImagesPullingTime="18.411281894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:00.510442253 +0000 UTC m=+21.687754324" watchObservedRunningTime="2026-09-08 01:59:00.533922623 +0000 UTC m=+21.711234696" Sep 08 01:59:01.313135 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.313108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:01.313235 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.313140 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:01.313285 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:01.313259 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:01.313392 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:01.313371 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:01.361603 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.361580 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 01:59:01.420550 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.420438 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T01:59:01.361598864Z","UUID":"14660852-61e1-4184-9bb1-75c7604d1275","Handler":null,"Name":"","Endpoint":""} Sep 08 01:59:01.422214 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.422195 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 01:59:01.422353 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.422228 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 01:59:01.442355 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.442316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" event={"ID":"2ca4f623-fdca-483b-b1e1-e0719251c2ab","Type":"ContainerStarted","Data":"1657695c39152f7d3f1ea2274fb34659cc9cecf92a7a1f73de567e28ba7125cd"} Sep 08 01:59:01.446047 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.445970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"72b29ee079c0a0d7177bca7d1e2471021f9e685b977b8d0ef53afeb2ac123376"} Sep 08 01:59:01.446181 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.446064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"ad2133d4f63b99abb14ec1a9b7a402780bda1d5a6cc60bd9bff062260558ab33"} Sep 08 01:59:01.446181 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.446113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"64476fcc6a46270409377c43453ae0b78a1c1b1199e737a9c16c21e8655ca60d"} Sep 08 01:59:01.446181 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.446146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"9abb11599dce38f6b48a9638b5bff2ec7dd565f51922d6628007dcb797aa6613"} Sep 08 01:59:01.446370 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.446184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"96aa21f2fcdf4049dc911ee60eb9eae90dac1f8491f89aa451ea0235d97aba2e"} Sep 08 01:59:01.446370 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:01.446228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"8475d8ae1bf27299fd3eee7b64b56880b8ae1dbe2dc7420ec8bfbdbd19a84200"} Sep 08 01:59:02.313046 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:02.312997 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:02.313170 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:02.313108 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:02.450705 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:02.450662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" event={"ID":"2ca4f623-fdca-483b-b1e1-e0719251c2ab","Type":"ContainerStarted","Data":"97954e388404768aa6672dd89fa7f362d821709cbca26f33bcd8145ad255ff65"} Sep 08 01:59:02.452428 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:02.452402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j965l" event={"ID":"72439b0a-1314-42c1-b32a-ecdea6839bb5","Type":"ContainerStarted","Data":"9a00f1875144c6103e6992fd73cb5ef8f19fb2baef926801cf9efd065c20e97c"} Sep 08 01:59:02.465815 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:02.465758 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4rh77" podStartSLOduration=2.202208981 podStartE2EDuration="22.465737814s" totalImagesPullingTime="20.263528833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:02.464922601 +0000 UTC m=+23.642234672" watchObservedRunningTime="2026-09-08 01:59:02.465737814 +0000 UTC m=+23.643049889" Sep 08 01:59:02.477742 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:02.477690 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j965l" podStartSLOduration=8.807101561 podStartE2EDuration="23.477673168s" totalImagesPullingTime="14.670571607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:02.477362409 +0000 UTC m=+23.654674483" watchObservedRunningTime="2026-09-08 01:59:02.477673168 +0000 UTC m=+23.654985240" Sep 08 01:59:03.313523 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:03.313489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:03.313679 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:03.313614 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:03.313761 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:03.313687 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:03.313880 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:03.313801 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:03.459213 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:03.459179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"d1643a9518f78938888481982d67b924074efd6c8a3ec2da6e00c541c08ea1b8"} Sep 08 01:59:03.854696 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:03.854655 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:59:03.855629 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:03.855602 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:59:04.313474 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:04.313145 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:04.313620 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:04.313517 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:05.313203 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.313160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:05.313203 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.313208 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:05.313737 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:05.313308 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:05.313737 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:05.313368 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:05.467826 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.467724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" event={"ID":"ec5768d3-add2-4b17-9b63-774b9810ffd0","Type":"ContainerStarted","Data":"419daba006791e4d97b2e2b31122346182af2238542a83ee8c58362c54561b4c"} Sep 08 01:59:05.468030 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.468011 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:05.469662 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.469633 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="65e5978118d73cae599d6278361a55ec7f6e37a9606f188782cad2fdcfa3880a" exitCode=0 Sep 08 01:59:05.469780 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.469665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"65e5978118d73cae599d6278361a55ec7f6e37a9606f188782cad2fdcfa3880a"} Sep 08 01:59:05.483561 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.483402 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:05.490781 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:05.490734 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" podStartSLOduration=7.7034426719999995 podStartE2EDuration="26.490720656s" totalImagesPullingTime="18.787277984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:05.490412618 +0000 UTC m=+26.667724689" watchObservedRunningTime="2026-09-08 01:59:05.490720656 +0000 UTC m=+26.668032728" Sep 08 01:59:06.313154 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.313124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:06.313320 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:06.313226 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:06.474198 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.474168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerStarted","Data":"aef37e8f28053a1b67bd9e26a56ab74f9b74de4fda932008ba0e57fbd3aa501d"} Sep 08 01:59:06.475789 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.475770 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:06.475893 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.475839 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:06.489159 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.489134 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:06.867248 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.867220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpgqg"] Sep 08 01:59:06.867406 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.867360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:06.867490 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:06.867465 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:06.869683 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.869659 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7c8t2"] Sep 08 01:59:06.869800 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.869754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:06.869876 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:06.869857 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:06.877758 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.877736 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9p5g4"] Sep 08 01:59:06.877834 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:06.877817 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:06.877909 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:06.877887 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:07.477933 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:07.477899 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="aef37e8f28053a1b67bd9e26a56ab74f9b74de4fda932008ba0e57fbd3aa501d" exitCode=0 Sep 08 01:59:07.478505 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:07.477984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"aef37e8f28053a1b67bd9e26a56ab74f9b74de4fda932008ba0e57fbd3aa501d"} Sep 08 01:59:08.042753 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.042714 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:59:08.042981 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.042854 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 01:59:08.043443 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.043418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhfwz" Sep 08 01:59:08.312958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.312929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:08.313055 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.312978 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:08.313055 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.313031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:08.313166 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:08.313047 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:08.313166 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:08.313150 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:08.313270 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:08.313231 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:08.481974 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.481890 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="6b6a9172980d62419128f5cb28920658147087b02c96f889dbd0b2bc78dd670e" exitCode=0 Sep 08 01:59:08.482469 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:08.481978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"6b6a9172980d62419128f5cb28920658147087b02c96f889dbd0b2bc78dd670e"} Sep 08 01:59:10.313268 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:10.313236 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:10.313876 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:10.313358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:10.313876 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:10.313391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:10.313876 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:10.313455 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:10.313876 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:10.313365 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:10.313876 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:10.313534 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:12.313204 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:12.313136 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:12.313677 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:12.313257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:12.313677 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:12.313271 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7c8t2" podUID="bf2486a7-aa44-4bbd-b047-262593d0ad76" Sep 08 01:59:12.313677 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:12.313319 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:12.313677 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:12.313390 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpgqg" podUID="d7c3ff89-2fe9-404a-8b67-54e343ca26e6" Sep 08 01:59:12.313677 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:12.313492 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9p5g4" podUID="526f198d-f102-4ef0-85dd-02a2399379ed" Sep 08 01:59:13.095240 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.095198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:13.095440 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.095364 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 01:59:13.095440 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.095437 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret podName:d7c3ff89-2fe9-404a-8b67-54e343ca26e6 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.095416998 +0000 UTC m=+66.272729078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret") pod "global-pull-secret-syncer-wpgqg" (UID: "d7c3ff89-2fe9-404a-8b67-54e343ca26e6") : object "kube-system"/"original-pull-secret" not registered Sep 08 01:59:13.151712 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.151684 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-162.ec2.internal" event="NodeReady" Sep 08 01:59:13.151881 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.151809 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 01:59:13.185348 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.185171 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 01:59:13.195654 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.195627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:13.195792 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.195663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:13.195792 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195765 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:59:13.195869 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195786 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 01:59:13.195869 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195805 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 01:59:13.195869 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195810 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs podName:bf2486a7-aa44-4bbd-b047-262593d0ad76 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.195797575 +0000 UTC m=+66.373109629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs") pod "network-metrics-daemon-7c8t2" (UID: "bf2486a7-aa44-4bbd-b047-262593d0ad76") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 01:59:13.195869 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195815 2559 projected.go:196] Error preparing data for projected volume kube-api-access-49tzs for pod openshift-network-diagnostics/network-check-target-9p5g4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:59:13.195869 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.195850 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs podName:526f198d-f102-4ef0-85dd-02a2399379ed nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.195839278 +0000 UTC m=+66.373151328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-49tzs" (UniqueName: "kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs") pod "network-check-target-9p5g4" (UID: "526f198d-f102-4ef0-85dd-02a2399379ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 01:59:13.206018 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.205997 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn"] Sep 08 01:59:13.206128 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.206110 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.211736 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.208244 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 01:59:13.211736 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.208547 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 01:59:13.211736 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.208608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zntsx\"" Sep 08 01:59:13.211736 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.208894 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 01:59:13.220560 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.220533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 01:59:13.226625 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.224407 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9"] Sep 08 01:59:13.226625 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.225234 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.227568 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.227448 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nf5vp\"" Sep 08 01:59:13.227709 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.227691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 01:59:13.227945 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.227895 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 01:59:13.228158 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.228141 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.228240 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.228210 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.242162 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.242140 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-q89gc"] Sep 08 01:59:13.242270 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.242247 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.244432 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.244412 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ccc49\"" Sep 08 01:59:13.244854 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.244834 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.245059 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.245045 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 01:59:13.245059 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.245056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 01:59:13.245188 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.245144 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.260508 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.260487 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl"] Sep 08 01:59:13.260602 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.260529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.262577 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.262560 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 01:59:13.262659 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.262629 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 01:59:13.262769 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.262732 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wmcp6\"" Sep 08 01:59:13.262902 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.262888 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.262975 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.262959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.267834 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.267817 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 01:59:13.272732 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.272715 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-65ff7fbcfb-55zh8"] Sep 08 01:59:13.272837 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.272824 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" Sep 08 01:59:13.274924 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.274907 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.275093 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.275080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wqz6b\"" Sep 08 01:59:13.275164 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.275119 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.289169 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.289108 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 01:59:13.289169 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.289163 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.289274 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.289194 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8j4cq"] Sep 08 01:59:13.291182 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291158 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 08 01:59:13.291370 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291200 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 08 01:59:13.291370 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291353 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hbl7z\"" Sep 08 01:59:13.291544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291421 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.291544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291438 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 08 01:59:13.291544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291443 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.291706 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.291558 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 08 01:59:13.295901 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.295882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.295993 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.295908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.295993 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.295925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.295993 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.295947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.296111 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.295993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk5c\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.296111 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.296111 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjpn\" (UniqueName: \"kubernetes.io/projected/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-kube-api-access-jxjpn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.296111 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.296111 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.296280 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.296280 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.296148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.305610 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.305592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.305688 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.305539 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xw95z"] Sep 08 01:59:13.309494 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.309471 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 01:59:13.309589 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.309547 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vtjdv\"" Sep 08 01:59:13.309659 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.309473 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.309714 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.309698 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 01:59:13.309767 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.309750 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.314218 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.314199 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 01:59:13.319899 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.319879 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6"] Sep 08 01:59:13.319985 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.319973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.322136 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.322122 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6zllf\"" Sep 08 01:59:13.322490 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.322472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 01:59:13.322648 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.322635 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 01:59:13.334414 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.334393 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-hjm2c"] Sep 08 01:59:13.334516 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.334503 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.336533 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.336514 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 08 01:59:13.336635 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.336520 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 08 01:59:13.336726 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.336707 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.336803 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.336790 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.336850 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.336815 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dhnlh\"" Sep 08 01:59:13.349038 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.349021 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677"] Sep 08 01:59:13.349128 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.349107 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.349201 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.349185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:13.351917 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.351866 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 08 01:59:13.351917 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.351887 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.352069 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.351948 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lxnlh\"" Sep 08 01:59:13.352069 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.352049 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.352181 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.352153 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xqzdk\"" Sep 08 01:59:13.352228 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.352218 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 08 01:59:13.356465 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.356441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:13.360953 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.360936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.363276 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.363249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 08 01:59:13.363276 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.363257 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.363455 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.363322 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qkztl\"" Sep 08 01:59:13.363499 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.363459 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.364609 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.364593 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rrx8x"] Sep 08 01:59:13.381328 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.381312 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8r5dv"] Sep 08 01:59:13.381444 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.381430 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.383630 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.383615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 01:59:13.383768 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.383615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r5fzh\"" Sep 08 01:59:13.383768 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.383619 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 01:59:13.383768 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.383704 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 01:59:13.394853 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394859 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-q89gc"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394915 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65ff7fbcfb-55zh8"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394932 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8j4cq"] Sep 08 01:59:13.394955 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394950 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xw95z"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrx8x"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.394996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.395014 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-hjm2c"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.395030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.395059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8r5dv"] Sep 08 01:59:13.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.395085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" Sep 08 01:59:13.396524 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.396595 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.396595 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-stats-auth\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.396704 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de250181-40c5-4498-9570-085aaaced35a-config-volume\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.396704 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396635 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rs8k\" (UniqueName: \"kubernetes.io/projected/041244a0-874b-4ed6-b34f-ad07dda18f5c-kube-api-access-6rs8k\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.396704 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.396808 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e7f2a9e-d218-4231-a95d-fd45d90c2625-config\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.396808 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4fs\" (UniqueName: \"kubernetes.io/projected/de250181-40c5-4498-9570-085aaaced35a-kube-api-access-6l4fs\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.396897 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjpn\" (UniqueName: \"kubernetes.io/projected/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-kube-api-access-jxjpn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.396897 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-tmp\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.397004 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk5c\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.397004 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.396924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8n2\" (UniqueName: \"kubernetes.io/projected/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-kube-api-access-fw8n2\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.397109 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e9f141c-f430-4cac-958d-841a7c761c57-serving-cert\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.397109 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397033 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l8rrs\"" Sep 08 01:59:13.397109 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.397488 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.398262 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.398365 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjmb\" (UniqueName: \"kubernetes.io/projected/dacd11f3-44cb-4220-b7f6-ac652c06f334-kube-api-access-nrjmb\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.398365 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.398466 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-snapshots\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.398466 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-trusted-ca\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.398466 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de250181-40c5-4498-9570-085aaaced35a-tmp-dir\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.398609 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.398609 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.398609 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hkh\" (UniqueName: \"kubernetes.io/projected/2354d2ac-5552-45b8-8228-4fb9bbaf2eff-kube-api-access-p9hkh\") pod \"volume-data-source-validator-695c895bd6-pb7fl\" (UID: \"2354d2ac-5552-45b8-8228-4fb9bbaf2eff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" Sep 08 01:59:13.398609 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7gqh\" (UniqueName: \"kubernetes.io/projected/0e9f141c-f430-4cac-958d-841a7c761c57-kube-api-access-c7gqh\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-default-certificate\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.398740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.398788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.399047 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.397895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.399349 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.399324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.399426 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.399364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.399850 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.399831 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:13.399911 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.399853 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:13.399955 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.399920 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:13.899902283 +0000 UTC m=+35.077214347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:13.400071 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.400113 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.400161 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5554\" (UniqueName: \"kubernetes.io/projected/4e7f2a9e-d218-4231-a95d-fd45d90c2625-kube-api-access-m5554\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.400208 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacd11f3-44cb-4220-b7f6-ac652c06f334-serving-cert\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.400257 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7f2a9e-d218-4231-a95d-fd45d90c2625-serving-cert\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.400334 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.400334 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.400310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-config\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.403727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.403175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.403727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.403450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.404502 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.404478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk5c\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.404660 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.404622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.405853 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.405833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjpn\" (UniqueName: \"kubernetes.io/projected/4ff518ec-91a9-4dcd-bd4c-62b4596db1ba-kube-api-access-jxjpn\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qbvhn\" (UID: \"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.407052 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.407032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.435330 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.435301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:13.500848 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh64k\" (UniqueName: \"kubernetes.io/projected/fb70bef4-078b-4963-8c98-de22fc5b1850-kube-api-access-jh64k\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.500848 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7f2a9e-d218-4231-a95d-fd45d90c2625-serving-cert\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.500996 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.500996 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-config\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.500996 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.500996 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-stats-auth\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.500998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ee1da86-5399-4b71-9b2e-bcf1509942ed-cert\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/3ee1da86-5399-4b71-9b2e-bcf1509942ed-kube-api-access-q99cb\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de250181-40c5-4498-9570-085aaaced35a-config-volume\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rs8k\" (UniqueName: \"kubernetes.io/projected/041244a0-874b-4ed6-b34f-ad07dda18f5c-kube-api-access-6rs8k\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.501187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e7f2a9e-d218-4231-a95d-fd45d90c2625-config\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4fs\" (UniqueName: \"kubernetes.io/projected/de250181-40c5-4498-9570-085aaaced35a-kube-api-access-6l4fs\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-tmp\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-nginx-conf\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8n2\" (UniqueName: \"kubernetes.io/projected/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-kube-api-access-fw8n2\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e9f141c-f430-4cac-958d-841a7c761c57-serving-cert\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjmb\" (UniqueName: \"kubernetes.io/projected/dacd11f3-44cb-4220-b7f6-ac652c06f334-kube-api-access-nrjmb\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-snapshots\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-trusted-ca\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de250181-40c5-4498-9570-085aaaced35a-tmp-dir\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.501544 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hkh\" (UniqueName: \"kubernetes.io/projected/2354d2ac-5552-45b8-8228-4fb9bbaf2eff-kube-api-access-p9hkh\") pod \"volume-data-source-validator-695c895bd6-pb7fl\" (UID: \"2354d2ac-5552-45b8-8228-4fb9bbaf2eff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501597 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk292\" (UniqueName: \"kubernetes.io/projected/df476139-f042-4119-a1ba-d4a2a3ab2e75-kube-api-access-mk292\") pod \"network-check-source-685456655f-8r5dv\" (UID: \"df476139-f042-4119-a1ba-d4a2a3ab2e75\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7gqh\" (UniqueName: \"kubernetes.io/projected/0e9f141c-f430-4cac-958d-841a7c761c57-kube-api-access-c7gqh\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-default-certificate\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-config\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501684 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de250181-40c5-4498-9570-085aaaced35a-config-volume\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5554\" (UniqueName: \"kubernetes.io/projected/4e7f2a9e-d218-4231-a95d-fd45d90c2625-kube-api-access-m5554\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacd11f3-44cb-4220-b7f6-ac652c06f334-serving-cert\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.501965 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.501768 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:13.502092 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.502048 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.002029744 +0000 UTC m=+35.179341800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:13.502765 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.502273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-snapshots\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.502765 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.501782 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:13.502765 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.502358 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:13.502765 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.502633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacd11f3-44cb-4220-b7f6-ac652c06f334-trusted-ca\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.502961 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.502920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de250181-40c5-4498-9570-085aaaced35a-tmp-dir\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.503023 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.502981 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.00296483 +0000 UTC m=+35.180276906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:13.503088 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.503045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9f141c-f430-4cac-958d-841a7c761c57-tmp\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.503088 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.503055 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.003040858 +0000 UTC m=+35.180352937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:13.503195 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.503118 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.003101662 +0000 UTC m=+35.180413717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:13.503582 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.503549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e7f2a9e-d218-4231-a95d-fd45d90c2625-config\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.503685 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.503642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f141c-f430-4cac-958d-841a7c761c57-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.503971 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.503939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7f2a9e-d218-4231-a95d-fd45d90c2625-serving-cert\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.504172 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.504148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-stats-auth\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.504711 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.504690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacd11f3-44cb-4220-b7f6-ac652c06f334-serving-cert\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.504807 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.504793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-default-certificate\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.505031 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.505016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e9f141c-f430-4cac-958d-841a7c761c57-serving-cert\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.511463 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.511434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rs8k\" (UniqueName: \"kubernetes.io/projected/041244a0-874b-4ed6-b34f-ad07dda18f5c-kube-api-access-6rs8k\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:13.511682 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.511665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hkh\" (UniqueName: \"kubernetes.io/projected/2354d2ac-5552-45b8-8228-4fb9bbaf2eff-kube-api-access-p9hkh\") pod \"volume-data-source-validator-695c895bd6-pb7fl\" (UID: \"2354d2ac-5552-45b8-8228-4fb9bbaf2eff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" Sep 08 01:59:13.513492 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.513467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7gqh\" (UniqueName: \"kubernetes.io/projected/0e9f141c-f430-4cac-958d-841a7c761c57-kube-api-access-c7gqh\") pod \"insights-operator-76bcd6ccf5-q89gc\" (UID: \"0e9f141c-f430-4cac-958d-841a7c761c57\") " pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.514206 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.514189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4fs\" (UniqueName: \"kubernetes.io/projected/de250181-40c5-4498-9570-085aaaced35a-kube-api-access-6l4fs\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:13.514721 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.514706 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5554\" (UniqueName: \"kubernetes.io/projected/4e7f2a9e-d218-4231-a95d-fd45d90c2625-kube-api-access-m5554\") pod \"service-ca-operator-859f94c4b-k9lp6\" (UID: \"4e7f2a9e-d218-4231-a95d-fd45d90c2625\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.515155 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.515131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8n2\" (UniqueName: \"kubernetes.io/projected/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-kube-api-access-fw8n2\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:13.515727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.515705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjmb\" (UniqueName: \"kubernetes.io/projected/dacd11f3-44cb-4220-b7f6-ac652c06f334-kube-api-access-nrjmb\") pod \"console-operator-85b454874b-8j4cq\" (UID: \"dacd11f3-44cb-4220-b7f6-ac652c06f334\") " pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.536037 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.536019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" Sep 08 01:59:13.568854 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.568833 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" Sep 08 01:59:13.579713 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.579697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" Sep 08 01:59:13.603099 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603081 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ee1da86-5399-4b71-9b2e-bcf1509942ed-cert\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/3ee1da86-5399-4b71-9b2e-bcf1509942ed-kube-api-access-q99cb\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-nginx-conf\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.603238 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.603315 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.103278435 +0000 UTC m=+35.280590487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:13.603323 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.603319 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:13.603847 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk292\" (UniqueName: \"kubernetes.io/projected/df476139-f042-4119-a1ba-d4a2a3ab2e75-kube-api-access-mk292\") pod \"network-check-source-685456655f-8r5dv\" (UID: \"df476139-f042-4119-a1ba-d4a2a3ab2e75\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" Sep 08 01:59:13.603847 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.603384 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.103368608 +0000 UTC m=+35.280680682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:13.603847 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh64k\" (UniqueName: \"kubernetes.io/projected/fb70bef4-078b-4963-8c98-de22fc5b1850-kube-api-access-jh64k\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.603999 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.603976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-nginx-conf\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:13.605863 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.605825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ee1da86-5399-4b71-9b2e-bcf1509942ed-cert\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.611593 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.611566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk292\" (UniqueName: \"kubernetes.io/projected/df476139-f042-4119-a1ba-d4a2a3ab2e75-kube-api-access-mk292\") pod \"network-check-source-685456655f-8r5dv\" (UID: \"df476139-f042-4119-a1ba-d4a2a3ab2e75\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" Sep 08 01:59:13.611891 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.611869 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/3ee1da86-5399-4b71-9b2e-bcf1509942ed-kube-api-access-q99cb\") pod \"ingress-canary-rrx8x\" (UID: \"3ee1da86-5399-4b71-9b2e-bcf1509942ed\") " pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.611977 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.611956 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh64k\" (UniqueName: \"kubernetes.io/projected/fb70bef4-078b-4963-8c98-de22fc5b1850-kube-api-access-jh64k\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:13.618704 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.618640 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:13.642114 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.642092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" Sep 08 01:59:13.688747 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.688707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrx8x" Sep 08 01:59:13.717479 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.717362 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" Sep 08 01:59:13.906024 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.906000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:13.906169 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.906151 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:13.906219 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.906171 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:13.906253 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:13.906226 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:14.906208502 +0000 UTC m=+36.083520571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:13.943763 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.943737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl"] Sep 08 01:59:13.955388 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.955356 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8j4cq"] Sep 08 01:59:13.956739 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.956630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn"] Sep 08 01:59:13.958064 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.958045 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-q89gc"] Sep 08 01:59:13.967620 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:13.967596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6"] Sep 08 01:59:14.002031 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.002011 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrx8x"] Sep 08 01:59:14.005627 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:14.005599 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff518ec_91a9_4dcd_bd4c_62b4596db1ba.slice/crio-950960dbcde224d8d4073d211179bbb87c79586ff4be4ad69eb1bb27cba285fe WatchSource:0}: Error finding container 950960dbcde224d8d4073d211179bbb87c79586ff4be4ad69eb1bb27cba285fe: Status 404 returned error can't find the container with id 950960dbcde224d8d4073d211179bbb87c79586ff4be4ad69eb1bb27cba285fe Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.006753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.006835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.006866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.006909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007091 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:14.007427 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007146 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.007127876 +0000 UTC m=+36.184439940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:14.007738 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007558 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.007540249 +0000 UTC m=+36.184852312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:14.007738 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007631 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:14.007738 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007666 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.007654667 +0000 UTC m=+36.184966732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:14.007738 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007720 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:14.008012 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.007749 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.007738699 +0000 UTC m=+36.185050765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:14.107724 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.107699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:14.107838 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.107736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:14.107878 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.107854 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:14.107923 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.107913 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.107899081 +0000 UTC m=+36.285211130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:14.107969 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.107852 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:14.108005 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.107990 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:15.107978909 +0000 UTC m=+36.285290972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:14.265856 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.265788 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8r5dv"] Sep 08 01:59:14.269023 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:14.268995 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf476139_f042_4119_a1ba_d4a2a3ab2e75.slice/crio-b8403a68ae1269822772950dac676720e8bbc69c08e033df484fe0c18696965a WatchSource:0}: Error finding container b8403a68ae1269822772950dac676720e8bbc69c08e033df484fe0c18696965a: Status 404 returned error can't find the container with id b8403a68ae1269822772950dac676720e8bbc69c08e033df484fe0c18696965a Sep 08 01:59:14.313510 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.313487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:14.313603 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.313492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:14.315468 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.315447 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 01:59:14.315785 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.315453 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 01:59:14.315785 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.315523 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2v8c\"" Sep 08 01:59:14.496090 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.496056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" event={"ID":"dacd11f3-44cb-4220-b7f6-ac652c06f334","Type":"ContainerStarted","Data":"3f1479532ab25a3dc4067f285c1552e169ae27bffa96e78287c1c111b62c695b"} Sep 08 01:59:14.497152 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.497129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" event={"ID":"0e9f141c-f430-4cac-958d-841a7c761c57","Type":"ContainerStarted","Data":"22df4c23a3ff826011d8084bfdc251fdd3b648d238dd852cbdeda98251846fea"} Sep 08 01:59:14.498165 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.498140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrx8x" event={"ID":"3ee1da86-5399-4b71-9b2e-bcf1509942ed","Type":"ContainerStarted","Data":"fb4312b01f47abfd6e60bcc56158c2132e60b60de14f9ab17b859a3c4209251a"} Sep 08 01:59:14.500913 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.500892 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="0e85754aa3edac49724d003a5ee2e86d6fe94cc04836869fdd0311fbd0b2dd51" exitCode=0 Sep 08 01:59:14.500988 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.500964 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"0e85754aa3edac49724d003a5ee2e86d6fe94cc04836869fdd0311fbd0b2dd51"} Sep 08 01:59:14.502093 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.502068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" event={"ID":"4e7f2a9e-d218-4231-a95d-fd45d90c2625","Type":"ContainerStarted","Data":"0f859db999755be50ec26a9d957fe3beb7e1933fe56227d149051f12b0ed981a"} Sep 08 01:59:14.503208 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.503190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" event={"ID":"2354d2ac-5552-45b8-8228-4fb9bbaf2eff","Type":"ContainerStarted","Data":"12751a431f8a0d45e5e557cfda6686093ed192816d28eed61eb3eac1f527edaf"} Sep 08 01:59:14.504275 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.504255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" event={"ID":"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba","Type":"ContainerStarted","Data":"950960dbcde224d8d4073d211179bbb87c79586ff4be4ad69eb1bb27cba285fe"} Sep 08 01:59:14.505322 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.505284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" event={"ID":"df476139-f042-4119-a1ba-d4a2a3ab2e75","Type":"ContainerStarted","Data":"b8403a68ae1269822772950dac676720e8bbc69c08e033df484fe0c18696965a"} Sep 08 01:59:14.916614 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:14.916055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:14.916614 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.916465 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:14.916614 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.916484 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:14.916614 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:14.916543 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:16.9165238 +0000 UTC m=+38.093835867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:15.017396 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.017358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:15.017564 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.017495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:15.017635 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.017564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:15.017635 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.017592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.017736 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.017802 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.017782709 +0000 UTC m=+38.195094773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.018187 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.018234 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.018219409 +0000 UTC m=+38.195531477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.018319 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.018308211 +0000 UTC m=+38.195620280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.018378 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:15.018453 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.018412 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.018401383 +0000 UTC m=+38.195713446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.119119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.119175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.119313 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.119376 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.11935743 +0000 UTC m=+38.296669499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.119438 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:15.119850 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:15.119471 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:17.119460747 +0000 UTC m=+38.296772812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:15.526112 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.526078 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f1566da-bfb8-430f-9d27-4f096b4ec427" containerID="d94225587a85c5e89109d5855822e2220a447dd58da0cb5f29cf9418605dd441" exitCode=0 Sep 08 01:59:15.526993 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:15.526160 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerDied","Data":"d94225587a85c5e89109d5855822e2220a447dd58da0cb5f29cf9418605dd441"} Sep 08 01:59:16.535276 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:16.535009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" event={"ID":"1f1566da-bfb8-430f-9d27-4f096b4ec427","Type":"ContainerStarted","Data":"bede28b08c688fde1a67fe1074da05d8d53042f35755dc710651b35bba993667"} Sep 08 01:59:16.555663 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:16.555362 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9jpxh" podStartSLOduration=6.961794416 podStartE2EDuration="36.55533679s" totalImagesPullingTime="29.593542374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:16.553551924 +0000 UTC m=+37.730863999" watchObservedRunningTime="2026-09-08 01:59:16.55533679 +0000 UTC m=+37.732648863" Sep 08 01:59:16.671997 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:16.671864 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:16.729018 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:16.728984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:16.937393 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:16.937309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:16.937540 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:16.937479 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:16.937540 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:16.937500 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:16.937637 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:16.937571 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:20.937554286 +0000 UTC m=+42.114866349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.038413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.038459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038576 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.038578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038633 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.038614888 +0000 UTC m=+42.215926954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038688 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.038718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038764 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.038745885 +0000 UTC m=+42.216057941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038828 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.038819824 +0000 UTC m=+42.216131903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038831 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:17.038852 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.038864 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.038853808 +0000 UTC m=+42.216165862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:17.139914 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.139877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:17.140091 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.140053 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:17.140150 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:17.140087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:17.140150 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.140136 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.14010675 +0000 UTC m=+42.317418846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:17.140262 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.140232 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:17.140335 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:17.140306 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:21.140276825 +0000 UTC m=+42.317588894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:18.341071 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:18.341031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:18.440567 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:18.440528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:18.489064 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:18.489034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:19.928040 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:19.927997 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:20.028529 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:20.028489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:20.974155 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:20.974118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:20.974663 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:20.974308 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:20.974663 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:20.974332 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:20.974663 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:20.974415 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:28.974392679 +0000 UTC m=+50.151704729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:21.074763 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.074728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:21.075007 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.074854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:21.075007 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.074888 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:21.075007 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.074928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:21.075007 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.074964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:21.075007 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.074981 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.074960008 +0000 UTC m=+50.252272070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:21.075243 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.075029 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.075017827 +0000 UTC m=+50.252329877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:21.075243 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.075049 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:21.075243 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.075055 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:21.075243 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.075088 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.075077203 +0000 UTC m=+50.252389280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:21.075243 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.075104 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.075095615 +0000 UTC m=+50.252407666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:21.176472 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.176434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:21.176692 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:21.176480 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:21.176692 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.176583 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:21.176692 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.176595 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:21.176692 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.176641 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.1766236 +0000 UTC m=+50.353935653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:21.176692 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:21.176663 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:29.176649968 +0000 UTC m=+50.353962018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:22.388211 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.388122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:22.557319 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.557191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" event={"ID":"4e7f2a9e-d218-4231-a95d-fd45d90c2625","Type":"ContainerStarted","Data":"b44bf660f4c06dc9c5bd940e6c2330c42ace5cda431303414955c3c4c868133b"} Sep 08 01:59:22.559079 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.559052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" event={"ID":"2354d2ac-5552-45b8-8228-4fb9bbaf2eff","Type":"ContainerStarted","Data":"b383b132138c05ec56750da4a84574857ac09bc4ac38a7c846317fecbeb027b1"} Sep 08 01:59:22.561154 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.561130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" event={"ID":"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba","Type":"ContainerStarted","Data":"8d280ed2b972179f8b9cd421cefaa7a6574d2c2d84be4234add48298284f706c"} Sep 08 01:59:22.562967 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.562943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" event={"ID":"df476139-f042-4119-a1ba-d4a2a3ab2e75","Type":"ContainerStarted","Data":"9167e1ff3f4285f6d2a43bb04d63a20999208993c83ee997ad6010d6fe5a8c8b"} Sep 08 01:59:22.564908 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.564888 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/0.log" Sep 08 01:59:22.565003 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.564924 2559 generic.go:356] "Generic (PLEG): container finished" podID="dacd11f3-44cb-4220-b7f6-ac652c06f334" containerID="ac025de96c765582a1dc568db51faea4f2e6316891c7cc4b613614cd8403972f" exitCode=255 Sep 08 01:59:22.565003 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.564983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" event={"ID":"dacd11f3-44cb-4220-b7f6-ac652c06f334","Type":"ContainerDied","Data":"ac025de96c765582a1dc568db51faea4f2e6316891c7cc4b613614cd8403972f"} Sep 08 01:59:22.565233 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.565212 2559 scope.go:117] "RemoveContainer" containerID="ac025de96c765582a1dc568db51faea4f2e6316891c7cc4b613614cd8403972f" Sep 08 01:59:22.567019 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.566697 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" event={"ID":"0e9f141c-f430-4cac-958d-841a7c761c57","Type":"ContainerStarted","Data":"4b65a975ab95b7639ed215aed3c0b0c920cfbc043e1bdcfbde998d5c6a803fb5"} Sep 08 01:59:22.568163 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.568135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrx8x" event={"ID":"3ee1da86-5399-4b71-9b2e-bcf1509942ed","Type":"ContainerStarted","Data":"f26b9077b303abe5de95b01541286e74aa5e844ed300ae7d72e4542c4c125569"} Sep 08 01:59:22.571979 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.571842 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" podStartSLOduration=27.697641441000002 podStartE2EDuration="35.571825879s" totalImagesPullingTime="7.874184438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.570642009 +0000 UTC m=+43.747954082" watchObservedRunningTime="2026-09-08 01:59:22.571825879 +0000 UTC m=+43.749137949" Sep 08 01:59:22.586016 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.585966 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-8r5dv" podStartSLOduration=27.958569044 podStartE2EDuration="35.585949822s" totalImagesPullingTime="7.627380778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.584066179 +0000 UTC m=+43.761378252" watchObservedRunningTime="2026-09-08 01:59:22.585949822 +0000 UTC m=+43.763261893" Sep 08 01:59:22.605311 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.605215 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" podStartSLOduration=28.731806337 podStartE2EDuration="36.605197977s" totalImagesPullingTime="7.87339164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.603629224 +0000 UTC m=+43.780941295" watchObservedRunningTime="2026-09-08 01:59:22.605197977 +0000 UTC m=+43.782510050" Sep 08 01:59:22.620965 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.620922 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" podStartSLOduration=28.748793618 podStartE2EDuration="36.620905303s" totalImagesPullingTime="7.872111685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.620203628 +0000 UTC m=+43.797515702" watchObservedRunningTime="2026-09-08 01:59:22.620905303 +0000 UTC m=+43.798217376" Sep 08 01:59:22.650628 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.650556 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rrx8x" podStartSLOduration=1.7785236260000001 podStartE2EDuration="9.650541999s" totalImagesPullingTime="7.872018373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.649397491 +0000 UTC m=+43.826709564" watchObservedRunningTime="2026-09-08 01:59:22.650541999 +0000 UTC m=+43.827854071" Sep 08 01:59:22.666192 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.665862 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-pb7fl" podStartSLOduration=28.930915361 podStartE2EDuration="36.665833968s" totalImagesPullingTime="7.734918607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:22.664549426 +0000 UTC m=+43.841861499" watchObservedRunningTime="2026-09-08 01:59:22.665833968 +0000 UTC m=+43.843146042" Sep 08 01:59:22.763036 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.762996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:22.863235 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:22.863195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:23.475545 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.475515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:23.571411 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.571366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:23.573518 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.573498 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 01:59:23.573974 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.573959 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/0.log" Sep 08 01:59:23.574040 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.573993 2559 generic.go:356] "Generic (PLEG): container finished" podID="dacd11f3-44cb-4220-b7f6-ac652c06f334" containerID="15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655" exitCode=255 Sep 08 01:59:23.574087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.574049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" event={"ID":"dacd11f3-44cb-4220-b7f6-ac652c06f334","Type":"ContainerDied","Data":"15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655"} Sep 08 01:59:23.574140 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.574109 2559 scope.go:117] "RemoveContainer" containerID="ac025de96c765582a1dc568db51faea4f2e6316891c7cc4b613614cd8403972f" Sep 08 01:59:23.574711 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.574314 2559 scope.go:117] "RemoveContainer" containerID="15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655" Sep 08 01:59:23.574711 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:23.574520 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8j4cq_openshift-console-operator(dacd11f3-44cb-4220-b7f6-ac652c06f334)\"" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" podUID="dacd11f3-44cb-4220-b7f6-ac652c06f334" Sep 08 01:59:23.619221 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.619199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:23.619328 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.619240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:23.671941 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.671922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:23.772085 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:23.772012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:24.097087 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.097012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:24.315617 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.315585 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" limit=1024 Sep 08 01:59:24.579375 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.579274 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 01:59:24.579718 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.579668 2559 scope.go:117] "RemoveContainer" containerID="15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655" Sep 08 01:59:24.579842 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:24.579821 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8j4cq_openshift-console-operator(dacd11f3-44cb-4220-b7f6-ac652c06f334)\"" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" podUID="dacd11f3-44cb-4220-b7f6-ac652c06f334" Sep 08 01:59:24.720162 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.720127 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4vp99"] Sep 08 01:59:24.770796 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.770761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4vp99"] Sep 08 01:59:24.770947 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.770888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:24.773555 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.773531 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xc9ld\"" Sep 08 01:59:24.773555 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.773540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 01:59:24.773555 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.773544 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 01:59:24.910504 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.910417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:24.910504 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.910475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/160555fc-4c7d-45ff-b071-235080ad81a4-crio-socket\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:24.910504 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.910491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/160555fc-4c7d-45ff-b071-235080ad81a4-kube-api-access-w4bj4\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:24.910786 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.910633 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/160555fc-4c7d-45ff-b071-235080ad81a4-data-volume\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:24.910786 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:24.910732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011605 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011752 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011752 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/160555fc-4c7d-45ff-b071-235080ad81a4-crio-socket\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011752 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/160555fc-4c7d-45ff-b071-235080ad81a4-kube-api-access-w4bj4\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011752 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.011705 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 01:59:25.011752 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.011737 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 01:59:25.011997 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.011765 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:25.511747329 +0000 UTC m=+46.689059398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : configmap "kube-rbac-proxy" not found Sep 08 01:59:25.011997 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/160555fc-4c7d-45ff-b071-235080ad81a4-data-volume\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.011997 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.011797 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:25.511780041 +0000 UTC m=+46.689092096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : secret "insights-runtime-extractor-tls" not found Sep 08 01:59:25.011997 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.011883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/160555fc-4c7d-45ff-b071-235080ad81a4-crio-socket\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.012187 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.012100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/160555fc-4c7d-45ff-b071-235080ad81a4-data-volume\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.020871 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.020848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bj4\" (UniqueName: \"kubernetes.io/projected/160555fc-4c7d-45ff-b071-235080ad81a4-kube-api-access-w4bj4\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.517791 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.517752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.517958 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.517824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:25.517958 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.517933 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 01:59:25.517958 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.517945 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 01:59:25.518112 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.518021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.518003253 +0000 UTC m=+47.695315317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : configmap "kube-rbac-proxy" not found Sep 08 01:59:25.518112 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.518040 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.518031929 +0000 UTC m=+47.695344002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : secret "insights-runtime-extractor-tls" not found Sep 08 01:59:25.582738 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:25.582710 2559 scope.go:117] "RemoveContainer" containerID="15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655" Sep 08 01:59:25.583082 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:25.582856 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8j4cq_openshift-console-operator(dacd11f3-44cb-4220-b7f6-ac652c06f334)\"" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" podUID="dacd11f3-44cb-4220-b7f6-ac652c06f334" Sep 08 01:59:26.528975 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:26.528938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:26.529150 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:26.529043 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 01:59:26.529150 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:26.529098 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:28.529082925 +0000 UTC m=+49.706394979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : configmap "kube-rbac-proxy" not found Sep 08 01:59:26.529150 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:26.529049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:26.529150 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:26.529130 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 01:59:26.529326 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:26.529185 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:28.529172918 +0000 UTC m=+49.706484967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : secret "insights-runtime-extractor-tls" not found Sep 08 01:59:27.089842 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:27.089811 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5496l_082e2a56-3930-4c0a-99b7-910f08509976/dns-node-resolver/0.log" Sep 08 01:59:28.288842 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:28.288815 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmcsm_128d5c0b-cee0-460b-a941-64eb1bd1c95a/node-ca/0.log" Sep 08 01:59:28.546342 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:28.546262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:28.546477 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:28.546358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:28.546477 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:28.546410 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 01:59:28.546477 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:28.546424 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 01:59:28.546477 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:28.546470 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:32.546456721 +0000 UTC m=+53.723768777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : configmap "kube-rbac-proxy" not found Sep 08 01:59:28.546611 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:28.546484 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:32.546478072 +0000 UTC m=+53.723790126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : secret "insights-runtime-extractor-tls" not found Sep 08 01:59:28.888939 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:28.888869 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrx8x_3ee1da86-5399-4b71-9b2e-bcf1509942ed/serve-healthcheck-canary/0.log" Sep 08 01:59:29.050793 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.050762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:29.050968 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.050919 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 01:59:29.050968 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.050940 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7566c6ff7c-b69xk: secret "image-registry-tls" not found Sep 08 01:59:29.051072 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.051000 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls podName:5544e5a7-a5c8-4d50-a59e-41b9eb6cec07 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.050981316 +0000 UTC m=+66.228293368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls") pod "image-registry-7566c6ff7c-b69xk" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07") : secret "image-registry-tls" not found Sep 08 01:59:29.151833 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.151805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:29.151989 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.151886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:29.151989 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.151914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:29.151989 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.151954 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.151938518 +0000 UTC m=+66.329250572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : configmap references non-existent config key: service-ca.crt Sep 08 01:59:29.152128 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.152002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:29.152128 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152019 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:29.152128 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152021 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 01:59:29.152128 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152076 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls podName:9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.152056948 +0000 UTC m=+66.329368999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxsr9" (UID: "9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba") : secret "cluster-monitoring-operator-tls" not found Sep 08 01:59:29.152128 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152097 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls podName:de250181-40c5-4498-9570-085aaaced35a nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.152087635 +0000 UTC m=+66.329399686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls") pod "dns-default-xw95z" (UID: "de250181-40c5-4498-9570-085aaaced35a") : secret "dns-default-metrics-tls" not found Sep 08 01:59:29.152329 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152143 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 01:59:29.152329 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.152187 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs podName:041244a0-874b-4ed6-b34f-ad07dda18f5c nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.152178981 +0000 UTC m=+66.329491066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs") pod "router-default-65ff7fbcfb-55zh8" (UID: "041244a0-874b-4ed6-b34f-ad07dda18f5c") : secret "router-metrics-certs-default" not found Sep 08 01:59:29.252857 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.252828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:29.252971 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.252862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:29.252971 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.252962 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 01:59:29.253039 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.252996 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls podName:fb70bef4-078b-4963-8c98-de22fc5b1850 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.252986041 +0000 UTC m=+66.430298104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-rc677" (UID: "fb70bef4-078b-4963-8c98-de22fc5b1850") : secret "samples-operator-tls" not found Sep 08 01:59:29.253039 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.252959 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 01:59:29.253039 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:29.253023 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert podName:1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc nodeName:}" failed. No retries permitted until 2026-09-08 01:59:45.253017708 +0000 UTC m=+66.430329770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-hjm2c" (UID: "1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc") : secret "networking-console-plugin-cert" not found Sep 08 01:59:29.490678 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:29.490611 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-qbvhn_4ff518ec-91a9-4dcd-bd4c-62b4596db1ba/kube-storage-version-migrator-operator/0.log" Sep 08 01:59:32.581792 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:32.581756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:32.582244 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:32.581871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:32.582244 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:32.581989 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 01:59:32.582244 ip-10-0-132-162 kubenswrapper[2559]: E0908 01:59:32.582038 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls podName:160555fc-4c7d-45ff-b071-235080ad81a4 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:40.582025235 +0000 UTC m=+61.759337286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vp99" (UID: "160555fc-4c7d-45ff-b071-235080ad81a4") : secret "insights-runtime-extractor-tls" not found Sep 08 01:59:32.582395 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:32.582352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/160555fc-4c7d-45ff-b071-235080ad81a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:38.493205 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:38.493176 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5hj6f" Sep 08 01:59:40.313010 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.312976 2559 scope.go:117] "RemoveContainer" containerID="15279a224a0b2deae11ece518b7efcdeb3cff52f6ac6fab90a261ce6255f9655" Sep 08 01:59:40.629409 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.629333 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 01:59:40.629409 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.629389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" event={"ID":"dacd11f3-44cb-4220-b7f6-ac652c06f334","Type":"ContainerStarted","Data":"e6ea27fe4b5febf3f467ce5f3c14571edd8fa8bb5d4f375da35d824a98a8c87a"} Sep 08 01:59:40.629695 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.629669 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:40.645978 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.645929 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" podStartSLOduration=45.774699268 podStartE2EDuration="53.645914862s" totalImagesPullingTime="7.871215594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:40.644412127 +0000 UTC m=+61.821724198" watchObservedRunningTime="2026-09-08 01:59:40.645914862 +0000 UTC m=+61.823226934" Sep 08 01:59:40.651133 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.651111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:40.653307 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.653267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/160555fc-4c7d-45ff-b071-235080ad81a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vp99\" (UID: \"160555fc-4c7d-45ff-b071-235080ad81a4\") " pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:40.682052 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.682031 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xc9ld\"" Sep 08 01:59:40.690687 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.690670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4vp99" Sep 08 01:59:40.719610 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.719581 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8j4cq" Sep 08 01:59:40.972747 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:40.972715 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4vp99"] Sep 08 01:59:40.976038 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:40.976009 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160555fc_4c7d_45ff_b071_235080ad81a4.slice/crio-fdbc40910108242c8cc16f483eccf57e0372a90cca8875ed93f1b449315bbd16 WatchSource:0}: Error finding container fdbc40910108242c8cc16f483eccf57e0372a90cca8875ed93f1b449315bbd16: Status 404 returned error can't find the container with id fdbc40910108242c8cc16f483eccf57e0372a90cca8875ed93f1b449315bbd16 Sep 08 01:59:41.634361 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:41.634318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vp99" event={"ID":"160555fc-4c7d-45ff-b071-235080ad81a4","Type":"ContainerStarted","Data":"76ac2a95bb33537657a051afff9978b4991b4a26d59ba09960e9611f486fb112"} Sep 08 01:59:41.634727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:41.634380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vp99" event={"ID":"160555fc-4c7d-45ff-b071-235080ad81a4","Type":"ContainerStarted","Data":"fdbc40910108242c8cc16f483eccf57e0372a90cca8875ed93f1b449315bbd16"} Sep 08 01:59:42.639854 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:42.639822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vp99" event={"ID":"160555fc-4c7d-45ff-b071-235080ad81a4","Type":"ContainerStarted","Data":"35fabf112e90c1c4266e9758670d1f3fb497aa4155158ed82032bb6f1d636bce"} Sep 08 01:59:43.645219 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:43.645182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vp99" event={"ID":"160555fc-4c7d-45ff-b071-235080ad81a4","Type":"ContainerStarted","Data":"7c7dd059d05ac6802cf0d0304063cf117ebe2d1d4913876613e7a2fd9d50bb01"} Sep 08 01:59:43.660860 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:43.660816 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4vp99" podStartSLOduration=17.612760248 podStartE2EDuration="19.660801058s" totalImagesPullingTime="2.04804081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:59:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:43.659713555 +0000 UTC m=+64.837025628" watchObservedRunningTime="2026-09-08 01:59:43.660801058 +0000 UTC m=+64.838113130" Sep 08 01:59:45.090600 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.090555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:45.092889 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.092859 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"image-registry-7566c6ff7c-b69xk\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:45.191307 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.191254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:45.191307 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.191311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:45.191512 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.191330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:45.191512 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.191353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:45.191512 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.191437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:45.192070 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.192045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041244a0-874b-4ed6-b34f-ad07dda18f5c-service-ca-bundle\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:45.193727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.193697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de250181-40c5-4498-9570-085aaaced35a-metrics-tls\") pod \"dns-default-xw95z\" (UID: \"de250181-40c5-4498-9570-085aaaced35a\") " pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:45.193727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.193717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxsr9\" (UID: \"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:45.193867 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.193756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/041244a0-874b-4ed6-b34f-ad07dda18f5c-metrics-certs\") pod \"router-default-65ff7fbcfb-55zh8\" (UID: \"041244a0-874b-4ed6-b34f-ad07dda18f5c\") " pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:45.193867 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.193771 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 01:59:45.204140 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.204120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7c3ff89-2fe9-404a-8b67-54e343ca26e6-original-pull-secret\") pod \"global-pull-secret-syncer-wpgqg\" (UID: \"d7c3ff89-2fe9-404a-8b67-54e343ca26e6\") " pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:45.223810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.223793 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpgqg" Sep 08 01:59:45.292790 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.292760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:45.292891 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.292804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:45.292891 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.292842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:45.292891 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.292876 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:45.295207 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.295183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb70bef4-078b-4963-8c98-de22fc5b1850-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-rc677\" (UID: \"fb70bef4-078b-4963-8c98-de22fc5b1850\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:45.295207 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.295197 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 01:59:45.295545 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.295527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-hjm2c\" (UID: \"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:45.295674 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.295659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tzs\" (UniqueName: \"kubernetes.io/projected/526f198d-f102-4ef0-85dd-02a2399379ed-kube-api-access-49tzs\") pod \"network-check-target-9p5g4\" (UID: \"526f198d-f102-4ef0-85dd-02a2399379ed\") " pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:45.305611 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.305587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf2486a7-aa44-4bbd-b047-262593d0ad76-metrics-certs\") pod \"network-metrics-daemon-7c8t2\" (UID: \"bf2486a7-aa44-4bbd-b047-262593d0ad76\") " pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:45.324857 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.324838 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zntsx\"" Sep 08 01:59:45.333477 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.333461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:45.352319 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.352282 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ccc49\"" Sep 08 01:59:45.361162 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.361144 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" Sep 08 01:59:45.399109 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.399084 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hbl7z\"" Sep 08 01:59:45.407852 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.407810 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:45.429839 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.429809 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6zllf\"" Sep 08 01:59:45.437932 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.437908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:45.462392 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.461589 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xqzdk\"" Sep 08 01:59:45.465403 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.465373 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lxnlh\"" Sep 08 01:59:45.469707 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.469684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" Sep 08 01:59:45.472418 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.472246 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qkztl\"" Sep 08 01:59:45.473527 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.473504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:45.480731 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.480391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" Sep 08 01:59:45.490373 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.490322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpgqg"] Sep 08 01:59:45.496897 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.496822 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c3ff89_2fe9_404a_8b67_54e343ca26e6.slice/crio-c20b50f5e26ecf1bbf36b86fc4800b382867d8ac516715b27ff7ae0405321cce WatchSource:0}: Error finding container c20b50f5e26ecf1bbf36b86fc4800b382867d8ac516715b27ff7ae0405321cce: Status 404 returned error can't find the container with id c20b50f5e26ecf1bbf36b86fc4800b382867d8ac516715b27ff7ae0405321cce Sep 08 01:59:45.530572 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.530344 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2v8c\"" Sep 08 01:59:45.538678 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.538642 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7c8t2" Sep 08 01:59:45.619877 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.619801 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 01:59:45.644912 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.644709 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9"] Sep 08 01:59:45.649336 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.649101 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9535a6aa_c1fe_4a1a_bcc1_ecabdfbb5bba.slice/crio-de2211e667ce83b5e97e997fb6acc0fc9abb9fc54680a50017cfac01ea2e9f1f WatchSource:0}: Error finding container de2211e667ce83b5e97e997fb6acc0fc9abb9fc54680a50017cfac01ea2e9f1f: Status 404 returned error can't find the container with id de2211e667ce83b5e97e997fb6acc0fc9abb9fc54680a50017cfac01ea2e9f1f Sep 08 01:59:45.652788 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.652757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" event={"ID":"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07","Type":"ContainerStarted","Data":"4213371027f6135fafc6a187c87276166ed010060f91a6fca519b7dd0bcd0fe7"} Sep 08 01:59:45.654900 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.654847 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpgqg" event={"ID":"d7c3ff89-2fe9-404a-8b67-54e343ca26e6","Type":"ContainerStarted","Data":"c20b50f5e26ecf1bbf36b86fc4800b382867d8ac516715b27ff7ae0405321cce"} Sep 08 01:59:45.697893 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.697865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65ff7fbcfb-55zh8"] Sep 08 01:59:45.699907 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.699882 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041244a0_874b_4ed6_b34f_ad07dda18f5c.slice/crio-d2037232f0d234dfb675a349fed354a89de061180223d383013ddb1058f488f9 WatchSource:0}: Error finding container d2037232f0d234dfb675a349fed354a89de061180223d383013ddb1058f488f9: Status 404 returned error can't find the container with id d2037232f0d234dfb675a349fed354a89de061180223d383013ddb1058f488f9 Sep 08 01:59:45.725653 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.725562 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xw95z"] Sep 08 01:59:45.727605 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.727582 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde250181_40c5_4498_9570_085aaaced35a.slice/crio-668d76b593ceaa5edbc676d36296c5f8822da0160eada09aa4aa19ed69bb3962 WatchSource:0}: Error finding container 668d76b593ceaa5edbc676d36296c5f8822da0160eada09aa4aa19ed69bb3962: Status 404 returned error can't find the container with id 668d76b593ceaa5edbc676d36296c5f8822da0160eada09aa4aa19ed69bb3962 Sep 08 01:59:45.772320 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.771825 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-hjm2c"] Sep 08 01:59:45.776329 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.776281 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf8d592_ea4b_4ba0_83a0_503d3fcc27bc.slice/crio-16439dbc2f9e038c407130f41535575e83596ceb1babb5b937d5f20092e254da WatchSource:0}: Error finding container 16439dbc2f9e038c407130f41535575e83596ceb1babb5b937d5f20092e254da: Status 404 returned error can't find the container with id 16439dbc2f9e038c407130f41535575e83596ceb1babb5b937d5f20092e254da Sep 08 01:59:45.777225 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.777194 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9p5g4"] Sep 08 01:59:45.780404 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.780369 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526f198d_f102_4ef0_85dd_02a2399379ed.slice/crio-abc84577a6f39dc38b0c5a40fbba2fdd53de4e92389c1c408b78ab93a1b4fae0 WatchSource:0}: Error finding container abc84577a6f39dc38b0c5a40fbba2fdd53de4e92389c1c408b78ab93a1b4fae0: Status 404 returned error can't find the container with id abc84577a6f39dc38b0c5a40fbba2fdd53de4e92389c1c408b78ab93a1b4fae0 Sep 08 01:59:45.788234 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.788211 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677"] Sep 08 01:59:45.841266 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:45.841240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7c8t2"] Sep 08 01:59:45.844254 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:45.844218 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2486a7_aa44_4bbd_b047_262593d0ad76.slice/crio-1227c4d13990e683b70a51c24048f8ff77f225048cee9a72c06ff96bddabf713 WatchSource:0}: Error finding container 1227c4d13990e683b70a51c24048f8ff77f225048cee9a72c06ff96bddabf713: Status 404 returned error can't find the container with id 1227c4d13990e683b70a51c24048f8ff77f225048cee9a72c06ff96bddabf713 Sep 08 01:59:46.664144 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.664015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" event={"ID":"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07","Type":"ContainerStarted","Data":"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c"} Sep 08 01:59:46.664613 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.664354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 01:59:46.668764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.667440 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9p5g4" event={"ID":"526f198d-f102-4ef0-85dd-02a2399379ed","Type":"ContainerStarted","Data":"c3650da4bd5d2dd4fc56cf2ae47d9c1af222049adf70da65b57f360782a16ff2"} Sep 08 01:59:46.668764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.667484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9p5g4" event={"ID":"526f198d-f102-4ef0-85dd-02a2399379ed","Type":"ContainerStarted","Data":"abc84577a6f39dc38b0c5a40fbba2fdd53de4e92389c1c408b78ab93a1b4fae0"} Sep 08 01:59:46.668764 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.667934 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 01:59:46.670897 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.670839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" event={"ID":"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc","Type":"ContainerStarted","Data":"16439dbc2f9e038c407130f41535575e83596ceb1babb5b937d5f20092e254da"} Sep 08 01:59:46.675810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.675783 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xw95z" event={"ID":"de250181-40c5-4498-9570-085aaaced35a","Type":"ContainerStarted","Data":"668d76b593ceaa5edbc676d36296c5f8822da0160eada09aa4aa19ed69bb3962"} Sep 08 01:59:46.679586 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.679559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7c8t2" event={"ID":"bf2486a7-aa44-4bbd-b047-262593d0ad76","Type":"ContainerStarted","Data":"1227c4d13990e683b70a51c24048f8ff77f225048cee9a72c06ff96bddabf713"} Sep 08 01:59:46.693834 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.691462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" event={"ID":"041244a0-874b-4ed6-b34f-ad07dda18f5c","Type":"ContainerStarted","Data":"668ac51f5387d40734f2d6c1bbfa1ae0a73d63021d7a7b6f0ea2082a1f0f6fd4"} Sep 08 01:59:46.693834 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.691538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" event={"ID":"041244a0-874b-4ed6-b34f-ad07dda18f5c","Type":"ContainerStarted","Data":"d2037232f0d234dfb675a349fed354a89de061180223d383013ddb1058f488f9"} Sep 08 01:59:46.693834 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.691760 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" podStartSLOduration=66.691741506 podStartE2EDuration="1m6.691741506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:46.690040793 +0000 UTC m=+67.867352865" watchObservedRunningTime="2026-09-08 01:59:46.691741506 +0000 UTC m=+67.869053579" Sep 08 01:59:46.698707 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.698677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" event={"ID":"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba","Type":"ContainerStarted","Data":"de2211e667ce83b5e97e997fb6acc0fc9abb9fc54680a50017cfac01ea2e9f1f"} Sep 08 01:59:46.702328 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.702302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" event={"ID":"fb70bef4-078b-4963-8c98-de22fc5b1850","Type":"ContainerStarted","Data":"1565e1a9818c907e15c69d69934597ca0ea2ac5fbb8e9aa1bc9a272baa81f4d9"} Sep 08 01:59:46.703523 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:46.703479 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9p5g4" podStartSLOduration=66.703464291 podStartE2EDuration="1m6.703464291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:46.702882909 +0000 UTC m=+67.880194983" watchObservedRunningTime="2026-09-08 01:59:46.703464291 +0000 UTC m=+67.880776361" Sep 08 01:59:47.409049 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:47.409007 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:47.412053 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:47.412025 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:47.431662 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:47.431607 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" podStartSLOduration=61.431586543 podStartE2EDuration="1m1.431586543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:46.728334778 +0000 UTC m=+67.905646853" watchObservedRunningTime="2026-09-08 01:59:47.431586543 +0000 UTC m=+68.608898616" Sep 08 01:59:47.708244 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:47.708162 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:47.709419 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:47.709397 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-65ff7fbcfb-55zh8" Sep 08 01:59:49.607895 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.607819 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 01:59:49.611369 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.611344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.613842 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.613817 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 01:59:49.613965 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.613935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 01:59:49.613965 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.613940 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 01:59:49.614619 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.614540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 01:59:49.614734 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.614635 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 01:59:49.614734 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.614637 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 01:59:49.614846 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.614743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9h72m\"" Sep 08 01:59:49.614846 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.614809 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 01:59:49.618271 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.618248 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 01:59:49.731262 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731225 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt9m\" (UniqueName: \"kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.731456 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.731456 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.731456 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.731588 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.731588 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.731576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832582 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.832816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.832767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt9m\" (UniqueName: \"kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.833388 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.833359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.833504 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.833439 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.833504 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.833475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.835461 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.835437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.835721 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.835704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.839810 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.839792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt9m\" (UniqueName: \"kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m\") pod \"console-5cfdbdb6d8-9xrrj\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:49.922739 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:49.922698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:51.047993 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.047920 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 01:59:51.125304 ip-10-0-132-162 kubenswrapper[2559]: W0908 01:59:51.125261 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea0c0c3_2674_4f83_875c_90e35572be60.slice/crio-04a8aae57f7aa3c05805ba5db72b5e7293a0ec36c5b8d1b07d877669640837c4 WatchSource:0}: Error finding container 04a8aae57f7aa3c05805ba5db72b5e7293a0ec36c5b8d1b07d877669640837c4: Status 404 returned error can't find the container with id 04a8aae57f7aa3c05805ba5db72b5e7293a0ec36c5b8d1b07d877669640837c4 Sep 08 01:59:51.728542 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.728505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7c8t2" event={"ID":"bf2486a7-aa44-4bbd-b047-262593d0ad76","Type":"ContainerStarted","Data":"a3a34c1ba12b3fc345f92993f08e9fefead10b723ce2e65a2e615cfeb79ba19b"} Sep 08 01:59:51.728792 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.728758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7c8t2" event={"ID":"bf2486a7-aa44-4bbd-b047-262593d0ad76","Type":"ContainerStarted","Data":"55be6b4bd6b8c49fb9894f7b9e19d4f177479fe265fb77a68fe7406cbf1c6369"} Sep 08 01:59:51.730607 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.730559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" event={"ID":"9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba","Type":"ContainerStarted","Data":"414c1e9a7d5268395458f76b02c2801d18cb4643670f1f06e77c9e2b653ba746"} Sep 08 01:59:51.732662 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.732637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdbdb6d8-9xrrj" event={"ID":"0ea0c0c3-2674-4f83-875c-90e35572be60","Type":"ContainerStarted","Data":"04a8aae57f7aa3c05805ba5db72b5e7293a0ec36c5b8d1b07d877669640837c4"} Sep 08 01:59:51.734421 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.734394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" event={"ID":"1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc","Type":"ContainerStarted","Data":"9053c7bbf607a7824ae921e8ad451ef32da5f4c7d5c99b0568db040c8dd02b60"} Sep 08 01:59:51.736727 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.736370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpgqg" event={"ID":"d7c3ff89-2fe9-404a-8b67-54e343ca26e6","Type":"ContainerStarted","Data":"8874c2b2799270a84678b83c849b36de2778bbe3bfed1fb766a22a410eedf2af"} Sep 08 01:59:51.738677 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.738654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" event={"ID":"fb70bef4-078b-4963-8c98-de22fc5b1850","Type":"ContainerStarted","Data":"1e57a7640035c64404588303b3de6ffe79192561fcb33651f705a628eb6719ec"} Sep 08 01:59:51.738783 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.738701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" event={"ID":"fb70bef4-078b-4963-8c98-de22fc5b1850","Type":"ContainerStarted","Data":"3abec2d639c8354dba1a09117bbeff2391e4676ad0e6b761c007459f4558beb2"} Sep 08 01:59:51.740816 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.740792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xw95z" event={"ID":"de250181-40c5-4498-9570-085aaaced35a","Type":"ContainerStarted","Data":"8d3ae265c1dacc95450aba27cbe338b1abd60ac2ceb449dd546ac173ddb911af"} Sep 08 01:59:51.740897 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.740838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xw95z" event={"ID":"de250181-40c5-4498-9570-085aaaced35a","Type":"ContainerStarted","Data":"aa93df5cde8b78a7216575d0c8653f174e10eef3bd32c39d338fd5972de693a1"} Sep 08 01:59:51.740951 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.740927 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xw95z" Sep 08 01:59:51.743839 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.743788 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7c8t2" podStartSLOduration=66.806851875 podStartE2EDuration="1m11.743770114s" totalImagesPullingTime="4.936918239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.74245424 +0000 UTC m=+72.919766312" watchObservedRunningTime="2026-09-08 01:59:51.743770114 +0000 UTC m=+72.921082186" Sep 08 01:59:51.758571 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.758508 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-rc677" podStartSLOduration=59.807095633 podStartE2EDuration="1m4.758487732s" totalImagesPullingTime="4.951392099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.756855413 +0000 UTC m=+72.934167499" watchObservedRunningTime="2026-09-08 01:59:51.758487732 +0000 UTC m=+72.935799805" Sep 08 01:59:51.777480 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.777419 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xw95z" podStartSLOduration=33.723849041 podStartE2EDuration="38.777401783s" totalImagesPullingTime="5.053552742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.776949331 +0000 UTC m=+72.954261404" watchObservedRunningTime="2026-09-08 01:59:51.777401783 +0000 UTC m=+72.954713855" Sep 08 01:59:51.791046 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.790992 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wpgqg" podStartSLOduration=66.135334727 podStartE2EDuration="1m11.790979561s" totalImagesPullingTime="5.655644834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.789823158 +0000 UTC m=+72.967135231" watchObservedRunningTime="2026-09-08 01:59:51.790979561 +0000 UTC m=+72.968291633" Sep 08 01:59:51.806478 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.806410 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxsr9" podStartSLOduration=60.669301934 podStartE2EDuration="1m5.80638782s" totalImagesPullingTime="5.137085886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.806044381 +0000 UTC m=+72.983356453" watchObservedRunningTime="2026-09-08 01:59:51.80638782 +0000 UTC m=+72.983699895" Sep 08 01:59:51.823147 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:51.823092 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-hjm2c" podStartSLOduration=39.819287332 podStartE2EDuration="44.823076565s" totalImagesPullingTime="5.003789233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:51.821798496 +0000 UTC m=+72.999110568" watchObservedRunningTime="2026-09-08 01:59:51.823076565 +0000 UTC m=+73.000388658" Sep 08 01:59:54.754557 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:54.754524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdbdb6d8-9xrrj" event={"ID":"0ea0c0c3-2674-4f83-875c-90e35572be60","Type":"ContainerStarted","Data":"f5e5e0defc12abca96cddab95b69b1daf740d0884e09756abab09f2771595457"} Sep 08 01:59:54.770644 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:54.770597 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cfdbdb6d8-9xrrj" podStartSLOduration=2.53393691 podStartE2EDuration="5.770582718s" totalImagesPullingTime="3.236645808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:54.770105528 +0000 UTC m=+75.947417590" watchObservedRunningTime="2026-09-08 01:59:54.770582718 +0000 UTC m=+75.947894790" Sep 08 01:59:59.923374 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:59.923339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:59.923827 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:59.923394 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 01:59:59.928252 ip-10-0-132-162 kubenswrapper[2559]: I0908 01:59:59.928229 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 02:00:00.075539 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.075504 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wvrsj"] Sep 08 02:00:00.079773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.079754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.081937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.081914 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 02:00:00.082046 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.081942 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 02:00:00.082046 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.081987 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 02:00:00.082455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.082434 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 02:00:00.082553 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.082515 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9gddq\"" Sep 08 02:00:00.216767 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-textfile\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.216767 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-tls\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.216767 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-wtmp\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s48kh\" (UniqueName: \"kubernetes.io/projected/38ad3666-a60c-4480-99e5-1e8da4df2c5e-kube-api-access-s48kh\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-sys\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-root\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216962 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-metrics-client-ca\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.217054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.216998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318249 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-wtmp\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s48kh\" (UniqueName: \"kubernetes.io/projected/38ad3666-a60c-4480-99e5-1e8da4df2c5e-kube-api-access-s48kh\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-sys\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-root\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-sys\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-wtmp\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-metrics-client-ca\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318576 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-textfile\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38ad3666-a60c-4480-99e5-1e8da4df2c5e-root\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-tls\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.318963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.318942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-textfile\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.319116 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.319096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-metrics-client-ca\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.319151 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.319096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.320623 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.320599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.320833 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.320815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38ad3666-a60c-4480-99e5-1e8da4df2c5e-node-exporter-tls\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.327239 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.327220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s48kh\" (UniqueName: \"kubernetes.io/projected/38ad3666-a60c-4480-99e5-1e8da4df2c5e-kube-api-access-s48kh\") pod \"node-exporter-wvrsj\" (UID: \"38ad3666-a60c-4480-99e5-1e8da4df2c5e\") " pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.388506 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.388475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wvrsj" Sep 08 02:00:00.399051 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:00:00.399029 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ad3666_a60c_4480_99e5_1e8da4df2c5e.slice/crio-268e61b731698ed043e0ab43ae8d59677ce3def551235164371f6a1f6124269c WatchSource:0}: Error finding container 268e61b731698ed043e0ab43ae8d59677ce3def551235164371f6a1f6124269c: Status 404 returned error can't find the container with id 268e61b731698ed043e0ab43ae8d59677ce3def551235164371f6a1f6124269c Sep 08 02:00:00.779651 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.779617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvrsj" event={"ID":"38ad3666-a60c-4480-99e5-1e8da4df2c5e","Type":"ContainerStarted","Data":"268e61b731698ed043e0ab43ae8d59677ce3def551235164371f6a1f6124269c"} Sep 08 02:00:00.783762 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:00.783739 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 02:00:01.748534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:01.748502 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xw95z" Sep 08 02:00:01.785110 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:01.785029 2559 generic.go:356] "Generic (PLEG): container finished" podID="38ad3666-a60c-4480-99e5-1e8da4df2c5e" containerID="6b9f598b0a35c0ae72473799f9a62fe541772f9edda561e8f10be7092233dec0" exitCode=0 Sep 08 02:00:01.785237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:01.785119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvrsj" event={"ID":"38ad3666-a60c-4480-99e5-1e8da4df2c5e","Type":"ContainerDied","Data":"6b9f598b0a35c0ae72473799f9a62fe541772f9edda561e8f10be7092233dec0"} Sep 08 02:00:02.791305 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:02.791262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvrsj" event={"ID":"38ad3666-a60c-4480-99e5-1e8da4df2c5e","Type":"ContainerStarted","Data":"abed7b8e2c3baf5916a1784157ec54108ae5b97fe2d6e5fd2caca2111ddd3698"} Sep 08 02:00:02.791667 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:02.791332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvrsj" event={"ID":"38ad3666-a60c-4480-99e5-1e8da4df2c5e","Type":"ContainerStarted","Data":"b6ac9910e4e1bf205b0fb82146e5b2f4b633a32c34759720e6ad9316530f1bc1"} Sep 08 02:00:02.809979 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:02.809926 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wvrsj" podStartSLOduration=1.641615262 podStartE2EDuration="2.809908705s" totalImagesPullingTime="1.108118048s" totalInitContainerRuntime="60.175395ms" isStatefulPod=true podCreationTimestamp="2026-09-08 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:00:02.80907578 +0000 UTC m=+83.986387851" watchObservedRunningTime="2026-09-08 02:00:02.809908705 +0000 UTC m=+83.987220774" Sep 08 02:00:07.711491 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:07.711461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 02:00:09.343470 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:09.343437 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 02:00:15.741511 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:15.741478 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 02:00:18.715184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:18.715150 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9p5g4" Sep 08 02:00:33.911384 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:33.911348 2559 generic.go:356] "Generic (PLEG): container finished" podID="0e9f141c-f430-4cac-958d-841a7c761c57" containerID="4b65a975ab95b7639ed215aed3c0b0c920cfbc043e1bdcfbde998d5c6a803fb5" exitCode=0 Sep 08 02:00:33.911789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:33.911423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" event={"ID":"0e9f141c-f430-4cac-958d-841a7c761c57","Type":"ContainerDied","Data":"4b65a975ab95b7639ed215aed3c0b0c920cfbc043e1bdcfbde998d5c6a803fb5"} Sep 08 02:00:33.911789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:33.911756 2559 scope.go:117] "RemoveContainer" containerID="4b65a975ab95b7639ed215aed3c0b0c920cfbc043e1bdcfbde998d5c6a803fb5" Sep 08 02:00:34.369028 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.368933 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" podUID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" containerName="registry" containerID="cri-o://d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c" gracePeriod=30 Sep 08 02:00:34.620016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.619958 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 02:00:34.683489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683462 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683500 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683527 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683560 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683586 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683607 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683865 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.683865 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.683691 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvk5c\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c\") pod \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\" (UID: \"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07\") " Sep 08 02:00:34.684237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.684088 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:00:34.684237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.684203 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-trusted-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.684560 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.684244 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:00:34.686203 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.686169 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:00:34.686392 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.686273 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:00:34.686392 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.686271 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:00:34.686528 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.686437 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:00:34.686668 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.686644 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c" (OuterVolumeSpecName: "kube-api-access-xvk5c") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "kube-api-access-xvk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:00:34.692091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.692065 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" (UID: "5544e5a7-a5c8-4d50-a59e-41b9eb6cec07"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:00:34.784746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784713 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784741 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-installation-pull-secrets\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784751 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-ca-trust-extracted\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784761 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk5c\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-kube-api-access-xvk5c\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784770 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-bound-sa-token\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784779 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-registry-certificates\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.784956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.784787 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07-image-registry-private-configuration\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:34.917263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.917225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-q89gc" event={"ID":"0e9f141c-f430-4cac-958d-841a7c761c57","Type":"ContainerStarted","Data":"f779d7a80797681a3316c746d5ce33dae68521068a28f61fa15651148d3af497"} Sep 08 02:00:34.918429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.918406 2559 generic.go:356] "Generic (PLEG): container finished" podID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" containerID="d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c" exitCode=0 Sep 08 02:00:34.918510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.918444 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" Sep 08 02:00:34.918510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.918461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" event={"ID":"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07","Type":"ContainerDied","Data":"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c"} Sep 08 02:00:34.918510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.918496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7566c6ff7c-b69xk" event={"ID":"5544e5a7-a5c8-4d50-a59e-41b9eb6cec07","Type":"ContainerDied","Data":"4213371027f6135fafc6a187c87276166ed010060f91a6fca519b7dd0bcd0fe7"} Sep 08 02:00:34.918623 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.918517 2559 scope.go:117] "RemoveContainer" containerID="d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c" Sep 08 02:00:34.926462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.926446 2559 scope.go:117] "RemoveContainer" containerID="d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c" Sep 08 02:00:34.926715 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:00:34.926694 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c\": container with ID starting with d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c not found: ID does not exist" containerID="d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c" Sep 08 02:00:34.926759 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.926723 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c"} err="failed to get container status \"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c\": rpc error: code = NotFound desc = could not find container \"d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c\": container with ID starting with d317f0221d476bbe89f84bc6e658c8f93a82d901db3ad0ecd460bb6f0a546c5c not found: ID does not exist" Sep 08 02:00:34.947897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.947876 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 02:00:34.956199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:34.956181 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7566c6ff7c-b69xk"] Sep 08 02:00:35.317987 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:35.317911 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" path="/var/lib/kubelet/pods/5544e5a7-a5c8-4d50-a59e-41b9eb6cec07/volumes" Sep 08 02:00:37.930997 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:37.930964 2559 generic.go:356] "Generic (PLEG): container finished" podID="4e7f2a9e-d218-4231-a95d-fd45d90c2625" containerID="b44bf660f4c06dc9c5bd940e6c2330c42ace5cda431303414955c3c4c868133b" exitCode=0 Sep 08 02:00:37.931370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:37.931020 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" event={"ID":"4e7f2a9e-d218-4231-a95d-fd45d90c2625","Type":"ContainerDied","Data":"b44bf660f4c06dc9c5bd940e6c2330c42ace5cda431303414955c3c4c868133b"} Sep 08 02:00:37.931370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:37.931311 2559 scope.go:117] "RemoveContainer" containerID="b44bf660f4c06dc9c5bd940e6c2330c42ace5cda431303414955c3c4c868133b" Sep 08 02:00:38.936778 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:38.936742 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-k9lp6" event={"ID":"4e7f2a9e-d218-4231-a95d-fd45d90c2625","Type":"ContainerStarted","Data":"4c3ff16c4829d6e7cd8e7ac180051ecdff9792d830e2034d9daa6dd1c283cf06"} Sep 08 02:00:40.768785 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:40.768740 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cfdbdb6d8-9xrrj" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerName="console" containerID="cri-o://f5e5e0defc12abca96cddab95b69b1daf740d0884e09756abab09f2771595457" gracePeriod=15 Sep 08 02:00:40.780330 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:40.780261 2559 patch_prober.go:28] interesting pod/console-5cfdbdb6d8-9xrrj container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Sep 08 02:00:40.780429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:40.780349 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5cfdbdb6d8-9xrrj" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" Sep 08 02:00:40.948080 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:40.948041 2559 generic.go:356] "Generic (PLEG): container finished" podID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerID="f5e5e0defc12abca96cddab95b69b1daf740d0884e09756abab09f2771595457" exitCode=0 Sep 08 02:00:40.948253 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:40.948120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdbdb6d8-9xrrj" event={"ID":"0ea0c0c3-2674-4f83-875c-90e35572be60","Type":"ContainerDied","Data":"f5e5e0defc12abca96cddab95b69b1daf740d0884e09756abab09f2771595457"} Sep 08 02:00:41.004162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.004131 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 02:00:41.130922 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.130825 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.130922 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.130895 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.131144 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.130925 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nt9m\" (UniqueName: \"kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.131144 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.130945 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.131144 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.130965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.131144 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.131001 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config\") pod \"0ea0c0c3-2674-4f83-875c-90e35572be60\" (UID: \"0ea0c0c3-2674-4f83-875c-90e35572be60\") " Sep 08 02:00:41.131468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.131438 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:00:41.131468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.131449 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:00:41.131468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.131442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config" (OuterVolumeSpecName: "console-config") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:00:41.133397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.133370 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:00:41.133503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.133468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m" (OuterVolumeSpecName: "kube-api-access-2nt9m") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "kube-api-access-2nt9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:00:41.133503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.133468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ea0c0c3-2674-4f83-875c-90e35572be60" (UID: "0ea0c0c3-2674-4f83-875c-90e35572be60"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:00:41.232098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232062 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-oauth-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.232098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232091 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt9m\" (UniqueName: \"kubernetes.io/projected/0ea0c0c3-2674-4f83-875c-90e35572be60-kube-api-access-2nt9m\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.232098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232101 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.232098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232109 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-oauth-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.232385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232117 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ea0c0c3-2674-4f83-875c-90e35572be60-console-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.232385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.232125 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ea0c0c3-2674-4f83-875c-90e35572be60-console-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:00:41.953096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.953058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdbdb6d8-9xrrj" event={"ID":"0ea0c0c3-2674-4f83-875c-90e35572be60","Type":"ContainerDied","Data":"04a8aae57f7aa3c05805ba5db72b5e7293a0ec36c5b8d1b07d877669640837c4"} Sep 08 02:00:41.953096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.953087 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdbdb6d8-9xrrj" Sep 08 02:00:41.953630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.953126 2559 scope.go:117] "RemoveContainer" containerID="f5e5e0defc12abca96cddab95b69b1daf740d0884e09756abab09f2771595457" Sep 08 02:00:41.966750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.966722 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 02:00:41.969839 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:41.969818 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cfdbdb6d8-9xrrj"] Sep 08 02:00:43.318684 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:43.318648 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" path="/var/lib/kubelet/pods/0ea0c0c3-2674-4f83-875c-90e35572be60/volumes" Sep 08 02:00:47.981980 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:47.981878 2559 generic.go:356] "Generic (PLEG): container finished" podID="4ff518ec-91a9-4dcd-bd4c-62b4596db1ba" containerID="8d280ed2b972179f8b9cd421cefaa7a6574d2c2d84be4234add48298284f706c" exitCode=0 Sep 08 02:00:47.981980 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:47.981913 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" event={"ID":"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba","Type":"ContainerDied","Data":"8d280ed2b972179f8b9cd421cefaa7a6574d2c2d84be4234add48298284f706c"} Sep 08 02:00:47.982402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:47.982204 2559 scope.go:117] "RemoveContainer" containerID="8d280ed2b972179f8b9cd421cefaa7a6574d2c2d84be4234add48298284f706c" Sep 08 02:00:48.987181 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:00:48.987149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qbvhn" event={"ID":"4ff518ec-91a9-4dcd-bd4c-62b4596db1ba","Type":"ContainerStarted","Data":"754445d7e3363a171dd6382c589f4e8514e81231f5630f8422bdc75ee11b104d"} Sep 08 02:01:24.339230 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339193 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7db6d78946-662wg"] Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339597 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerName="console" Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339614 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerName="console" Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339629 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" containerName="registry" Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339637 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" containerName="registry" Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339708 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea0c0c3-2674-4f83-875c-90e35572be60" containerName="console" Sep 08 02:01:24.339853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.339721 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5544e5a7-a5c8-4d50-a59e-41b9eb6cec07" containerName="registry" Sep 08 02:01:24.343052 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.343030 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.345081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 08 02:01:24.345253 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345238 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 08 02:01:24.345346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345258 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 08 02:01:24.345346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345258 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zj9nr\"" Sep 08 02:01:24.345346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345260 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 08 02:01:24.345476 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.345333 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 08 02:01:24.349992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.349974 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 08 02:01:24.353847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.353826 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7db6d78946-662wg"] Sep 08 02:01:24.455912 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.455884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.455920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.455944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.456042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g49d\" (UniqueName: \"kubernetes.io/projected/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-kube-api-access-4g49d\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.456061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456206 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.456083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-federate-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456206 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.456106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-metrics-client-ca\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.456206 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.456123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-serving-certs-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.556944 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.556911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g49d\" (UniqueName: \"kubernetes.io/projected/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-kube-api-access-4g49d\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.556952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-federate-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-metrics-client-ca\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-serving-certs-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557943 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-metrics-client-ca\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.557943 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.557929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-serving-certs-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.558381 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.558358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.559667 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.559643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-federate-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.559764 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.559680 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.559764 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.559711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.559764 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.559716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-telemeter-client-tls\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.563928 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.563905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g49d\" (UniqueName: \"kubernetes.io/projected/0cf31d75-ff1f-40ea-8e9d-fbbe0180392a-kube-api-access-4g49d\") pod \"telemeter-client-7db6d78946-662wg\" (UID: \"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a\") " pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.653233 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.653203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" Sep 08 02:01:24.927366 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:24.927274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7db6d78946-662wg"] Sep 08 02:01:24.931273 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:01:24.931240 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf31d75_ff1f_40ea_8e9d_fbbe0180392a.slice/crio-9608d4d44237c8698e17fbcc24a24ecdf2847012c60e5b4420512eac4d404310 WatchSource:0}: Error finding container 9608d4d44237c8698e17fbcc24a24ecdf2847012c60e5b4420512eac4d404310: Status 404 returned error can't find the container with id 9608d4d44237c8698e17fbcc24a24ecdf2847012c60e5b4420512eac4d404310 Sep 08 02:01:25.114375 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:25.114333 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" event={"ID":"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a","Type":"ContainerStarted","Data":"9608d4d44237c8698e17fbcc24a24ecdf2847012c60e5b4420512eac4d404310"} Sep 08 02:01:27.124993 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:27.124955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" event={"ID":"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a","Type":"ContainerStarted","Data":"7346dae4e99dd9d12ac9b6cb7972372dda6c3a6e125ad26576452304e0a49e22"} Sep 08 02:01:29.134411 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.134372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" event={"ID":"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a","Type":"ContainerStarted","Data":"3e3b7e65e16e91615e36f2ce2bf866453f143fe16b78b9812e7b2ee952fa1182"} Sep 08 02:01:29.134787 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.134430 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" event={"ID":"0cf31d75-ff1f-40ea-8e9d-fbbe0180392a","Type":"ContainerStarted","Data":"5e693a87c7c82a8d721abb304967a3f2c435ce112a427da2d34e641abe74189f"} Sep 08 02:01:29.154219 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.154167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7db6d78946-662wg" podStartSLOduration=1.7566989419999999 podStartE2EDuration="5.154150992s" totalImagesPullingTime="3.39745205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:01:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:01:29.152272779 +0000 UTC m=+170.329584865" watchObservedRunningTime="2026-09-08 02:01:29.154150992 +0000 UTC m=+170.331463064" Sep 08 02:01:29.229953 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.229921 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:01:29.233127 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.233100 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.235470 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.235434 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 02:01:29.235596 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.235524 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 02:01:29.235863 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.235832 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9h72m\"" Sep 08 02:01:29.235863 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.235851 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 02:01:29.236006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.235905 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 02:01:29.236198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.236179 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 02:01:29.236363 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.236342 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 02:01:29.236736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.236720 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 02:01:29.241065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.241012 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 02:01:29.245357 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.245330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:01:29.395765 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68v7k\" (UniqueName: \"kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.395924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.395914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496528 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496528 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68v7k\" (UniqueName: \"kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496799 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496799 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496799 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.496799 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.497020 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.496836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.497573 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.497544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.497692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.497602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.497692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.497646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.497692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.497657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.498963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.498944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.499156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.499142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.505042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.505019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68v7k\" (UniqueName: \"kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k\") pod \"console-fb5d9cf7d-k4z58\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.545924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.545878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:29.817091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:29.817056 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:01:29.820124 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:01:29.820103 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4234bce_705f_45b7_800d_00ece075fec3.slice/crio-d0092a6dbcf90a25b6d3765e341a793bf2ae9347b079e238343d8a6663bc6617 WatchSource:0}: Error finding container d0092a6dbcf90a25b6d3765e341a793bf2ae9347b079e238343d8a6663bc6617: Status 404 returned error can't find the container with id d0092a6dbcf90a25b6d3765e341a793bf2ae9347b079e238343d8a6663bc6617 Sep 08 02:01:30.139927 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:30.139830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb5d9cf7d-k4z58" event={"ID":"e4234bce-705f-45b7-800d-00ece075fec3","Type":"ContainerStarted","Data":"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6"} Sep 08 02:01:30.139927 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:30.139890 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb5d9cf7d-k4z58" event={"ID":"e4234bce-705f-45b7-800d-00ece075fec3","Type":"ContainerStarted","Data":"d0092a6dbcf90a25b6d3765e341a793bf2ae9347b079e238343d8a6663bc6617"} Sep 08 02:01:30.156473 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:30.156419 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fb5d9cf7d-k4z58" podStartSLOduration=1.156401178 podStartE2EDuration="1.156401178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:01:30.155676102 +0000 UTC m=+171.332988189" watchObservedRunningTime="2026-09-08 02:01:30.156401178 +0000 UTC m=+171.333713250" Sep 08 02:01:39.546717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:39.546631 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:39.547107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:39.546718 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:39.551504 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:39.551479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:40.179758 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.179733 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:01:40.331664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.331630 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:01:40.336551 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.336525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.345398 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.345374 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:01:40.372176 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372358 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372358 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372358 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrbd\" (UniqueName: \"kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.372483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.372449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473902 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473902 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrbd\" (UniqueName: \"kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.473902 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.473649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.474334 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.474306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.474441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.474413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.474501 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.474457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.474597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.474572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.476245 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.476224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.476370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.476319 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.482452 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.482423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrbd\" (UniqueName: \"kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd\") pod \"console-77547f95bc-gcv68\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.646516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.646477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:40.920842 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:40.920807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:01:40.923331 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:01:40.923302 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e99fa06_e18b_4029_a96d_1c7efda62529.slice/crio-ce368bb4eeefecca1bc65a68c64b7b908e8fa855b7db46589781de0929ffff60 WatchSource:0}: Error finding container ce368bb4eeefecca1bc65a68c64b7b908e8fa855b7db46589781de0929ffff60: Status 404 returned error can't find the container with id ce368bb4eeefecca1bc65a68c64b7b908e8fa855b7db46589781de0929ffff60 Sep 08 02:01:41.181458 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:41.181363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77547f95bc-gcv68" event={"ID":"1e99fa06-e18b-4029-a96d-1c7efda62529","Type":"ContainerStarted","Data":"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0"} Sep 08 02:01:41.181458 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:41.181415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77547f95bc-gcv68" event={"ID":"1e99fa06-e18b-4029-a96d-1c7efda62529","Type":"ContainerStarted","Data":"ce368bb4eeefecca1bc65a68c64b7b908e8fa855b7db46589781de0929ffff60"} Sep 08 02:01:41.196636 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:41.196592 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77547f95bc-gcv68" podStartSLOduration=1.196576397 podStartE2EDuration="1.196576397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:01:41.195546079 +0000 UTC m=+182.372858153" watchObservedRunningTime="2026-09-08 02:01:41.196576397 +0000 UTC m=+182.373888469" Sep 08 02:01:50.646579 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:50.646541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:50.647035 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:50.646608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:50.650984 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:50.650961 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:51.222871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:51.222844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:01:51.267084 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:01:51.267053 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:02:15.238765 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.238730 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h"] Sep 08 02:02:15.242096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.242079 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.244366 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.244338 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 02:02:15.244507 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.244373 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vqx5j\"" Sep 08 02:02:15.244507 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.244373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 02:02:15.249738 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.249716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h"] Sep 08 02:02:15.346498 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.346471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.346628 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.346520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdrq\" (UniqueName: \"kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.346628 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.346604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.447441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.447416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.447570 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.447447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.447570 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.447500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdrq\" (UniqueName: \"kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.447771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.447755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.447840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.447825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.454727 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.454705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdrq\" (UniqueName: \"kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.551803 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.551748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:15.821090 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:15.820995 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h"] Sep 08 02:02:15.823515 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:02:15.823488 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73abab6c_ba29_4699_a719_ca9238f53bef.slice/crio-f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf WatchSource:0}: Error finding container f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf: Status 404 returned error can't find the container with id f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf Sep 08 02:02:16.290414 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.290355 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fb5d9cf7d-k4z58" podUID="e4234bce-705f-45b7-800d-00ece075fec3" containerName="console" containerID="cri-o://4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6" gracePeriod=15 Sep 08 02:02:16.314571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.314529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" event={"ID":"73abab6c-ba29-4699-a719-ca9238f53bef","Type":"ContainerStarted","Data":"f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf"} Sep 08 02:02:16.530851 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.530828 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:02:16.657139 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68v7k\" (UniqueName: \"kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657189 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657241 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657470 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657315 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657470 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657483 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca\") pod \"e4234bce-705f-45b7-800d-00ece075fec3\" (UID: \"e4234bce-705f-45b7-800d-00ece075fec3\") " Sep 08 02:02:16.657788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657750 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:02:16.657788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657761 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:02:16.657936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config" (OuterVolumeSpecName: "console-config") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:02:16.657992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.657950 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca" (OuterVolumeSpecName: "service-ca") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:02:16.659420 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.659398 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:02:16.659813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.659793 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:02:16.659869 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.659815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k" (OuterVolumeSpecName: "kube-api-access-68v7k") pod "e4234bce-705f-45b7-800d-00ece075fec3" (UID: "e4234bce-705f-45b7-800d-00ece075fec3"). InnerVolumeSpecName "kube-api-access-68v7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:02:16.758964 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.758924 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-oauth-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.758964 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.758960 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-console-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.758964 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.758970 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.759197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.758979 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68v7k\" (UniqueName: \"kubernetes.io/projected/e4234bce-705f-45b7-800d-00ece075fec3-kube-api-access-68v7k\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.759197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.758992 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4234bce-705f-45b7-800d-00ece075fec3-console-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.759197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.759005 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-oauth-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:16.759197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:16.759018 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4234bce-705f-45b7-800d-00ece075fec3-trusted-ca-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:17.320680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.320643 2559 generic.go:356] "Generic (PLEG): container finished" podID="e4234bce-705f-45b7-800d-00ece075fec3" containerID="4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6" exitCode=0 Sep 08 02:02:17.321186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.320684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb5d9cf7d-k4z58" event={"ID":"e4234bce-705f-45b7-800d-00ece075fec3","Type":"ContainerDied","Data":"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6"} Sep 08 02:02:17.321186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.320700 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb5d9cf7d-k4z58" Sep 08 02:02:17.321186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.320729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb5d9cf7d-k4z58" event={"ID":"e4234bce-705f-45b7-800d-00ece075fec3","Type":"ContainerDied","Data":"d0092a6dbcf90a25b6d3765e341a793bf2ae9347b079e238343d8a6663bc6617"} Sep 08 02:02:17.321186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.320757 2559 scope.go:117] "RemoveContainer" containerID="4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6" Sep 08 02:02:17.330631 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.330605 2559 scope.go:117] "RemoveContainer" containerID="4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6" Sep 08 02:02:17.330974 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:02:17.330938 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6\": container with ID starting with 4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6 not found: ID does not exist" containerID="4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6" Sep 08 02:02:17.331079 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.330977 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6"} err="failed to get container status \"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6\": rpc error: code = NotFound desc = could not find container \"4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6\": container with ID starting with 4923e33d767a080bd4631583a7cfc2b138931563dcd80188ff5fdf76430846d6 not found: ID does not exist" Sep 08 02:02:17.344370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.344340 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:02:17.348081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:17.348056 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb5d9cf7d-k4z58"] Sep 08 02:02:19.319558 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:19.319526 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4234bce-705f-45b7-800d-00ece075fec3" path="/var/lib/kubelet/pods/e4234bce-705f-45b7-800d-00ece075fec3/volumes" Sep 08 02:02:22.345888 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:22.345795 2559 generic.go:356] "Generic (PLEG): container finished" podID="73abab6c-ba29-4699-a719-ca9238f53bef" containerID="1e5e81c33be74de532770631f9ef77a3012c6481db8e25a85a50a9bd3ecf9917" exitCode=0 Sep 08 02:02:22.345888 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:22.345846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" event={"ID":"73abab6c-ba29-4699-a719-ca9238f53bef","Type":"ContainerDied","Data":"1e5e81c33be74de532770631f9ef77a3012c6481db8e25a85a50a9bd3ecf9917"} Sep 08 02:02:25.359902 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:25.359870 2559 generic.go:356] "Generic (PLEG): container finished" podID="73abab6c-ba29-4699-a719-ca9238f53bef" containerID="1220bc6217f96b77c7d742797165101b6a3cc10ea3958edbb9bc7bee6db72564" exitCode=0 Sep 08 02:02:25.360311 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:25.359956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" event={"ID":"73abab6c-ba29-4699-a719-ca9238f53bef","Type":"ContainerDied","Data":"1220bc6217f96b77c7d742797165101b6a3cc10ea3958edbb9bc7bee6db72564"} Sep 08 02:02:32.392679 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:32.392645 2559 generic.go:356] "Generic (PLEG): container finished" podID="73abab6c-ba29-4699-a719-ca9238f53bef" containerID="7d601a02c5d224a3b9afca39805f7fa2d5795c618f2278166d09fa7c543226cf" exitCode=0 Sep 08 02:02:32.393099 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:32.392712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" event={"ID":"73abab6c-ba29-4699-a719-ca9238f53bef","Type":"ContainerDied","Data":"7d601a02c5d224a3b9afca39805f7fa2d5795c618f2278166d09fa7c543226cf"} Sep 08 02:02:33.516360 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.516337 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:33.697829 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.697804 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util\") pod \"73abab6c-ba29-4699-a719-ca9238f53bef\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " Sep 08 02:02:33.697998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.697834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle\") pod \"73abab6c-ba29-4699-a719-ca9238f53bef\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " Sep 08 02:02:33.697998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.697861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqdrq\" (UniqueName: \"kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq\") pod \"73abab6c-ba29-4699-a719-ca9238f53bef\" (UID: \"73abab6c-ba29-4699-a719-ca9238f53bef\") " Sep 08 02:02:33.698535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.698510 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle" (OuterVolumeSpecName: "bundle") pod "73abab6c-ba29-4699-a719-ca9238f53bef" (UID: "73abab6c-ba29-4699-a719-ca9238f53bef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:02:33.700092 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.700070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq" (OuterVolumeSpecName: "kube-api-access-tqdrq") pod "73abab6c-ba29-4699-a719-ca9238f53bef" (UID: "73abab6c-ba29-4699-a719-ca9238f53bef"). InnerVolumeSpecName "kube-api-access-tqdrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:02:33.701976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.701950 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util" (OuterVolumeSpecName: "util") pod "73abab6c-ba29-4699-a719-ca9238f53bef" (UID: "73abab6c-ba29-4699-a719-ca9238f53bef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:02:33.798709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.798663 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:33.798709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.798702 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73abab6c-ba29-4699-a719-ca9238f53bef-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:33.798709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:33.798712 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqdrq\" (UniqueName: \"kubernetes.io/projected/73abab6c-ba29-4699-a719-ca9238f53bef-kube-api-access-tqdrq\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:34.401815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:34.401771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" event={"ID":"73abab6c-ba29-4699-a719-ca9238f53bef","Type":"ContainerDied","Data":"f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf"} Sep 08 02:02:34.401986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:34.401837 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50e59bc6f9622b2fb24a7707aa943edb52b97af9179aa4a15001d718bc2c4cf" Sep 08 02:02:34.401986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:34.401788 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097s9h" Sep 08 02:02:38.168913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.168882 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc"] Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169150 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="util" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169160 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="util" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169180 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="pull" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169185 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="pull" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169194 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4234bce-705f-45b7-800d-00ece075fec3" containerName="console" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169202 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4234bce-705f-45b7-800d-00ece075fec3" containerName="console" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169208 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="extract" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169213 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="extract" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169260 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73abab6c-ba29-4699-a719-ca9238f53bef" containerName="extract" Sep 08 02:02:38.169268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.169267 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4234bce-705f-45b7-800d-00ece075fec3" containerName="console" Sep 08 02:02:38.210739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.210703 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc"] Sep 08 02:02:38.210885 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.210797 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.213249 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.213228 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 02:02:38.213385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.213229 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 02:02:38.213385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.213229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nk69l\"" Sep 08 02:02:38.225997 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.225976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmmg\" (UniqueName: \"kubernetes.io/projected/b0f61986-99f0-49e1-9674-5325c8441cb6-kube-api-access-tzmmg\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.226081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.226043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0f61986-99f0-49e1-9674-5325c8441cb6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.326616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.326579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0f61986-99f0-49e1-9674-5325c8441cb6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.326786 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.326627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmmg\" (UniqueName: \"kubernetes.io/projected/b0f61986-99f0-49e1-9674-5325c8441cb6-kube-api-access-tzmmg\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.326960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.326941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0f61986-99f0-49e1-9674-5325c8441cb6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.335226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.335198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmmg\" (UniqueName: \"kubernetes.io/projected/b0f61986-99f0-49e1-9674-5325c8441cb6-kube-api-access-tzmmg\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmlc\" (UID: \"b0f61986-99f0-49e1-9674-5325c8441cb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.519769 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.519690 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" Sep 08 02:02:38.809671 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:38.809594 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc"] Sep 08 02:02:38.815440 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:02:38.815401 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f61986_99f0_49e1_9674_5325c8441cb6.slice/crio-3c8e6214f9bab38126d978045d9378269d7cfa1f38a36c1911e54160e44d2e72 WatchSource:0}: Error finding container 3c8e6214f9bab38126d978045d9378269d7cfa1f38a36c1911e54160e44d2e72: Status 404 returned error can't find the container with id 3c8e6214f9bab38126d978045d9378269d7cfa1f38a36c1911e54160e44d2e72 Sep 08 02:02:39.420743 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:39.420702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" event={"ID":"b0f61986-99f0-49e1-9674-5325c8441cb6","Type":"ContainerStarted","Data":"3c8e6214f9bab38126d978045d9378269d7cfa1f38a36c1911e54160e44d2e72"} Sep 08 02:02:48.457304 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:48.457256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" event={"ID":"b0f61986-99f0-49e1-9674-5325c8441cb6","Type":"ContainerStarted","Data":"22f075e41e684bcf7cc49fc44a5079e87f9bfffe1a2d7c425f756c379657041b"} Sep 08 02:02:48.476547 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:48.476496 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmlc" podStartSLOduration=1.818654056 podStartE2EDuration="10.476479431s" totalImagesPullingTime="8.657825375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:02:48.475192687 +0000 UTC m=+249.652504760" watchObservedRunningTime="2026-09-08 02:02:48.476479431 +0000 UTC m=+249.653791506" Sep 08 02:02:50.215655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.215624 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs"] Sep 08 02:02:50.218837 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.218822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.221219 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.221189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 02:02:50.221592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.221564 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 02:02:50.221881 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.221857 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vqx5j\"" Sep 08 02:02:50.227284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.227260 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs"] Sep 08 02:02:50.320262 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.320227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8p5\" (UniqueName: \"kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.320410 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.320273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.320410 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.320319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.421601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.421573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8p5\" (UniqueName: \"kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.421757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.421614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.421757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.421632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.421952 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.421933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.422008 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.421961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.430196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.430175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8p5\" (UniqueName: \"kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.531979 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.531911 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:50.804196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:50.804001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs"] Sep 08 02:02:50.806867 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:02:50.806838 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6f4436_5ded_49ce_bcb5_03ab4d45055c.slice/crio-ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110 WatchSource:0}: Error finding container ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110: Status 404 returned error can't find the container with id ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110 Sep 08 02:02:51.471084 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:51.471052 2559 generic.go:356] "Generic (PLEG): container finished" podID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerID="109630662c72ddf859fb56ba29c1ac9bb095728673520f05d0d66335dd36d997" exitCode=0 Sep 08 02:02:51.471503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:51.471138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" event={"ID":"1a6f4436-5ded-49ce-bcb5-03ab4d45055c","Type":"ContainerDied","Data":"109630662c72ddf859fb56ba29c1ac9bb095728673520f05d0d66335dd36d997"} Sep 08 02:02:51.471503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:51.471184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" event={"ID":"1a6f4436-5ded-49ce-bcb5-03ab4d45055c","Type":"ContainerStarted","Data":"ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110"} Sep 08 02:02:55.490448 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:55.490421 2559 generic.go:356] "Generic (PLEG): container finished" podID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerID="d676a73680b839f00dbffc737fccac031d7d45a816f9e76047d98f0328a551f6" exitCode=0 Sep 08 02:02:55.490776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:55.490511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" event={"ID":"1a6f4436-5ded-49ce-bcb5-03ab4d45055c","Type":"ContainerDied","Data":"d676a73680b839f00dbffc737fccac031d7d45a816f9e76047d98f0328a551f6"} Sep 08 02:02:56.292398 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.292362 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz"] Sep 08 02:02:56.295506 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.295484 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.297566 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.297544 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ck5hn\"" Sep 08 02:02:56.298303 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.298273 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 02:02:56.298406 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.298276 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 02:02:56.303994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.303970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz"] Sep 08 02:02:56.367682 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.367653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g6t\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-kube-api-access-68g6t\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.367836 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.367704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.468346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.468312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g6t\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-kube-api-access-68g6t\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.468488 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.468361 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.475836 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.475809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.475978 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.475955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g6t\" (UniqueName: \"kubernetes.io/projected/a7992bf4-347d-4b9b-8b19-a9034f8a4768-kube-api-access-68g6t\") pod \"cert-manager-cainjector-559cdd8c87-zkwwz\" (UID: \"a7992bf4-347d-4b9b-8b19-a9034f8a4768\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.496215 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.496191 2559 generic.go:356] "Generic (PLEG): container finished" podID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerID="c2e055ecad62ead5d01a466c3dce12acf6712a927f7c4ec81995158699c9d144" exitCode=0 Sep 08 02:02:56.496523 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.496223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" event={"ID":"1a6f4436-5ded-49ce-bcb5-03ab4d45055c","Type":"ContainerDied","Data":"c2e055ecad62ead5d01a466c3dce12acf6712a927f7c4ec81995158699c9d144"} Sep 08 02:02:56.613354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.613282 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" Sep 08 02:02:56.881798 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:56.881772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz"] Sep 08 02:02:56.883483 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:02:56.883458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7992bf4_347d_4b9b_8b19_a9034f8a4768.slice/crio-34d39243d8f88fd5a87a31a186ee5b7c4bc0607c39b14fb59fbb7bbb008d8a29 WatchSource:0}: Error finding container 34d39243d8f88fd5a87a31a186ee5b7c4bc0607c39b14fb59fbb7bbb008d8a29: Status 404 returned error can't find the container with id 34d39243d8f88fd5a87a31a186ee5b7c4bc0607c39b14fb59fbb7bbb008d8a29 Sep 08 02:02:57.501904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.501865 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" event={"ID":"a7992bf4-347d-4b9b-8b19-a9034f8a4768","Type":"ContainerStarted","Data":"34d39243d8f88fd5a87a31a186ee5b7c4bc0607c39b14fb59fbb7bbb008d8a29"} Sep 08 02:02:57.640454 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.640429 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:57.679251 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.679210 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle\") pod \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " Sep 08 02:02:57.679251 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.679253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util\") pod \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " Sep 08 02:02:57.679487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.679352 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t8p5\" (UniqueName: \"kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5\") pod \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\" (UID: \"1a6f4436-5ded-49ce-bcb5-03ab4d45055c\") " Sep 08 02:02:57.679883 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.679852 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle" (OuterVolumeSpecName: "bundle") pod "1a6f4436-5ded-49ce-bcb5-03ab4d45055c" (UID: "1a6f4436-5ded-49ce-bcb5-03ab4d45055c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:02:57.682120 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.682091 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5" (OuterVolumeSpecName: "kube-api-access-2t8p5") pod "1a6f4436-5ded-49ce-bcb5-03ab4d45055c" (UID: "1a6f4436-5ded-49ce-bcb5-03ab4d45055c"). InnerVolumeSpecName "kube-api-access-2t8p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:02:57.685168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.685141 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util" (OuterVolumeSpecName: "util") pod "1a6f4436-5ded-49ce-bcb5-03ab4d45055c" (UID: "1a6f4436-5ded-49ce-bcb5-03ab4d45055c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:02:57.780260 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.780186 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:57.780260 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.780214 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:57.780260 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:57.780224 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8p5\" (UniqueName: \"kubernetes.io/projected/1a6f4436-5ded-49ce-bcb5-03ab4d45055c-kube-api-access-2t8p5\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:02:58.510703 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:58.510669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" event={"ID":"1a6f4436-5ded-49ce-bcb5-03ab4d45055c","Type":"ContainerDied","Data":"ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110"} Sep 08 02:02:58.511109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:58.510715 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkshzs" Sep 08 02:02:58.511109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:58.510720 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1d791b35729b6de4f33c3b2dc10002f1f6c1c4ef8e5f26587af1df85225110" Sep 08 02:02:59.217064 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217035 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s52kq"] Sep 08 02:02:59.217429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217411 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="util" Sep 08 02:02:59.217429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217429 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="util" Sep 08 02:02:59.217588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217457 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="pull" Sep 08 02:02:59.217588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217464 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="pull" Sep 08 02:02:59.217588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217472 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="extract" Sep 08 02:02:59.217588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217481 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="extract" Sep 08 02:02:59.217588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.217546 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6f4436-5ded-49ce-bcb5-03ab4d45055c" containerName="extract" Sep 08 02:02:59.221070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.221041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.223205 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.223182 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gr8tp\"" Sep 08 02:02:59.228003 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.227981 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s52kq"] Sep 08 02:02:59.294541 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.294511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7q5s\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-kube-api-access-k7q5s\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.294681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.294595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-bound-sa-token\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.395031 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.394993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7q5s\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-kube-api-access-k7q5s\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.395200 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.395099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-bound-sa-token\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.402543 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.402516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-bound-sa-token\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.402732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.402708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7q5s\" (UniqueName: \"kubernetes.io/projected/2df94578-e5cf-47b0-89c5-0ae90baddd61-kube-api-access-k7q5s\") pod \"cert-manager-55b4d954d5-s52kq\" (UID: \"2df94578-e5cf-47b0-89c5-0ae90baddd61\") " pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.531388 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.531321 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s52kq" Sep 08 02:02:59.810832 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:02:59.810759 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s52kq"] Sep 08 02:02:59.814985 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:02:59.814949 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2df94578_e5cf_47b0_89c5_0ae90baddd61.slice/crio-ce9277b83fc3461dd27b5dbde782ec5228b00d51dceb925b4acb2b42c4fa564c WatchSource:0}: Error finding container ce9277b83fc3461dd27b5dbde782ec5228b00d51dceb925b4acb2b42c4fa564c: Status 404 returned error can't find the container with id ce9277b83fc3461dd27b5dbde782ec5228b00d51dceb925b4acb2b42c4fa564c Sep 08 02:03:00.519897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:00.519863 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s52kq" event={"ID":"2df94578-e5cf-47b0-89c5-0ae90baddd61","Type":"ContainerStarted","Data":"a91de97ed3a286c0a20cad9ee84764115da9c87b542420534f194c347b0f208a"} Sep 08 02:03:00.519897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:00.519909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s52kq" event={"ID":"2df94578-e5cf-47b0-89c5-0ae90baddd61","Type":"ContainerStarted","Data":"ce9277b83fc3461dd27b5dbde782ec5228b00d51dceb925b4acb2b42c4fa564c"} Sep 08 02:03:00.521481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:00.521453 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" event={"ID":"a7992bf4-347d-4b9b-8b19-a9034f8a4768","Type":"ContainerStarted","Data":"c9eb13fbf3aade19b392b2e8ee179a0840671562bbffc2349e820e654613b131"} Sep 08 02:03:00.539136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:00.539090 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s52kq" podStartSLOduration=1.5390759410000001 podStartE2EDuration="1.539075941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:00.537490998 +0000 UTC m=+261.714803069" watchObservedRunningTime="2026-09-08 02:03:00.539075941 +0000 UTC m=+261.716388012" Sep 08 02:03:00.553000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:00.552957 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zkwwz" podStartSLOduration=1.88777396 podStartE2EDuration="4.552943727s" totalImagesPullingTime="2.665169767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:00.552008708 +0000 UTC m=+261.729320780" watchObservedRunningTime="2026-09-08 02:03:00.552943727 +0000 UTC m=+261.730255800" Sep 08 02:03:10.637636 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.637563 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d"] Sep 08 02:03:10.640824 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.640805 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.643111 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.643085 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 02:03:10.643111 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.643102 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 02:03:10.643284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.643125 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vqx5j\"" Sep 08 02:03:10.651179 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.651157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d"] Sep 08 02:03:10.790900 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.790852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtglj\" (UniqueName: \"kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.790900 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.790900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.790900 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.790921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.891681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.891587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtglj\" (UniqueName: \"kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.891681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.891627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.891681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.891645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.892016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.891954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.892016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.891990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.898521 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.898494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtglj\" (UniqueName: \"kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:10.950399 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:10.950370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:11.218504 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:11.218474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d"] Sep 08 02:03:11.221541 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:03:11.221501 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1f7945_aa13_4965_b707_069de885d832.slice/crio-784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7 WatchSource:0}: Error finding container 784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7: Status 404 returned error can't find the container with id 784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7 Sep 08 02:03:11.571600 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:11.571508 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a1f7945-aa13-4965-b707-069de885d832" containerID="9fdfb66aaca57cdcffb8260775facf843541cfcc586a5a9fcfa5a5f516487c8e" exitCode=0 Sep 08 02:03:11.571600 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:11.571563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" event={"ID":"8a1f7945-aa13-4965-b707-069de885d832","Type":"ContainerDied","Data":"9fdfb66aaca57cdcffb8260775facf843541cfcc586a5a9fcfa5a5f516487c8e"} Sep 08 02:03:11.571773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:11.571614 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" event={"ID":"8a1f7945-aa13-4965-b707-069de885d832","Type":"ContainerStarted","Data":"784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7"} Sep 08 02:03:12.578183 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:12.578088 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a1f7945-aa13-4965-b707-069de885d832" containerID="518a432b3590e392322c88674e700f318b70fa04b20a376ea36c45ca4175eeb9" exitCode=0 Sep 08 02:03:12.578557 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:12.578178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" event={"ID":"8a1f7945-aa13-4965-b707-069de885d832","Type":"ContainerDied","Data":"518a432b3590e392322c88674e700f318b70fa04b20a376ea36c45ca4175eeb9"} Sep 08 02:03:13.584561 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:13.584524 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a1f7945-aa13-4965-b707-069de885d832" containerID="61d931e0d7e7b9f78f2ce7983675c4f514e09f084d84901f389f3df0420a30e8" exitCode=0 Sep 08 02:03:13.584892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:13.584583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" event={"ID":"8a1f7945-aa13-4965-b707-069de885d832","Type":"ContainerDied","Data":"61d931e0d7e7b9f78f2ce7983675c4f514e09f084d84901f389f3df0420a30e8"} Sep 08 02:03:14.710924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.710902 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:14.821794 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.821763 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util\") pod \"8a1f7945-aa13-4965-b707-069de885d832\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " Sep 08 02:03:14.821901 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.821806 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle\") pod \"8a1f7945-aa13-4965-b707-069de885d832\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " Sep 08 02:03:14.821948 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.821911 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtglj\" (UniqueName: \"kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj\") pod \"8a1f7945-aa13-4965-b707-069de885d832\" (UID: \"8a1f7945-aa13-4965-b707-069de885d832\") " Sep 08 02:03:14.822610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.822580 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle" (OuterVolumeSpecName: "bundle") pod "8a1f7945-aa13-4965-b707-069de885d832" (UID: "8a1f7945-aa13-4965-b707-069de885d832"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:03:14.823913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.823891 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj" (OuterVolumeSpecName: "kube-api-access-jtglj") pod "8a1f7945-aa13-4965-b707-069de885d832" (UID: "8a1f7945-aa13-4965-b707-069de885d832"). InnerVolumeSpecName "kube-api-access-jtglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:03:14.827161 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.827142 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util" (OuterVolumeSpecName: "util") pod "8a1f7945-aa13-4965-b707-069de885d832" (UID: "8a1f7945-aa13-4965-b707-069de885d832"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:03:14.922880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.922856 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtglj\" (UniqueName: \"kubernetes.io/projected/8a1f7945-aa13-4965-b707-069de885d832-kube-api-access-jtglj\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:14.922880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.922878 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:14.923023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:14.922887 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a1f7945-aa13-4965-b707-069de885d832-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:15.595813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:15.595722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" event={"ID":"8a1f7945-aa13-4965-b707-069de885d832","Type":"ContainerDied","Data":"784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7"} Sep 08 02:03:15.595813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:15.595758 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxrb9d" Sep 08 02:03:15.595813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:15.595797 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="784ab409b25828fc7e35e4f415eec15c53b682ac3eb5b3c99d95e9f9d1b3e9c7" Sep 08 02:03:27.882266 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882230 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd"] Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882542 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="pull" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882553 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="pull" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882570 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="util" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882576 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="util" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882583 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="extract" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882588 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="extract" Sep 08 02:03:27.882739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.882635 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1f7945-aa13-4965-b707-069de885d832" containerName="extract" Sep 08 02:03:27.890110 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.890092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:27.892734 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.892712 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vqx5j\"" Sep 08 02:03:27.892903 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.892714 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 02:03:27.893519 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.893498 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 02:03:27.893650 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:27.893625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd"] Sep 08 02:03:28.028160 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.028116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bp2k\" (UniqueName: \"kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.028355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.028186 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.028355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.028235 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.108876 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.108837 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk"] Sep 08 02:03:28.112471 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.112453 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.116092 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.116068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 02:03:28.116540 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.116513 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 02:03:28.116628 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.116612 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 02:03:28.116709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.116691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzwkl\"" Sep 08 02:03:28.116756 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.116724 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 02:03:28.129023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.128999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.129123 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.129029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.129123 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.129076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bp2k\" (UniqueName: \"kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.129399 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.129383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.129458 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.129443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.137462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.137406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk"] Sep 08 02:03:28.141774 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.141743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bp2k\" (UniqueName: \"kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.200038 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.200001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:28.229876 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.229834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.230039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.229888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.230039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.229925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.230039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.229948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.230039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.229982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trt8\" (UniqueName: \"kubernetes.io/projected/577fd71c-6d67-4013-829a-055a9cddf509-kube-api-access-6trt8\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331038 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.330995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.331045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trt8\" (UniqueName: \"kubernetes.io/projected/577fd71c-6d67-4013-829a-055a9cddf509-kube-api-access-6trt8\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.331094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.331111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331407 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.331279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.331581 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.331560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.333364 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.333339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/577fd71c-6d67-4013-829a-055a9cddf509-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.333589 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.333573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.333788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.333767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/577fd71c-6d67-4013-829a-055a9cddf509-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.339879 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.339856 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trt8\" (UniqueName: \"kubernetes.io/projected/577fd71c-6d67-4013-829a-055a9cddf509-kube-api-access-6trt8\") pod \"opendatahub-operator-controller-manager-6566bd6ff-dcbvk\" (UID: \"577fd71c-6d67-4013-829a-055a9cddf509\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.421945 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.421899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:28.490815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.490784 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd"] Sep 08 02:03:28.492426 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:03:28.492394 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda553c8db_1713_4476_8c5d_ba8a34c76345.slice/crio-460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba WatchSource:0}: Error finding container 460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba: Status 404 returned error can't find the container with id 460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba Sep 08 02:03:28.655802 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.655768 2559 generic.go:356] "Generic (PLEG): container finished" podID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerID="670d94767b0fe6501bf639922acaf81d8105a67fa8cb12ee24592113c3b5e13a" exitCode=0 Sep 08 02:03:28.655968 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.655855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" event={"ID":"a553c8db-1713-4476-8c5d-ba8a34c76345","Type":"ContainerDied","Data":"670d94767b0fe6501bf639922acaf81d8105a67fa8cb12ee24592113c3b5e13a"} Sep 08 02:03:28.655968 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.655901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" event={"ID":"a553c8db-1713-4476-8c5d-ba8a34c76345","Type":"ContainerStarted","Data":"460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba"} Sep 08 02:03:28.700584 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:28.700543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk"] Sep 08 02:03:28.702834 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:03:28.702810 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577fd71c_6d67_4013_829a_055a9cddf509.slice/crio-a18003398f9a30c71089009fbe20ea31c4f8f97b932720550334890d524961d4 WatchSource:0}: Error finding container a18003398f9a30c71089009fbe20ea31c4f8f97b932720550334890d524961d4: Status 404 returned error can't find the container with id a18003398f9a30c71089009fbe20ea31c4f8f97b932720550334890d524961d4 Sep 08 02:03:29.661708 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:29.661673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" event={"ID":"577fd71c-6d67-4013-829a-055a9cddf509","Type":"ContainerStarted","Data":"a18003398f9a30c71089009fbe20ea31c4f8f97b932720550334890d524961d4"} Sep 08 02:03:29.663521 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:29.663497 2559 generic.go:356] "Generic (PLEG): container finished" podID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerID="c2d4fe7adc73e03e349966d99b1acc60503290a915cfa07372dc53d75255c316" exitCode=0 Sep 08 02:03:29.663647 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:29.663547 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" event={"ID":"a553c8db-1713-4476-8c5d-ba8a34c76345","Type":"ContainerDied","Data":"c2d4fe7adc73e03e349966d99b1acc60503290a915cfa07372dc53d75255c316"} Sep 08 02:03:30.671819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:30.671788 2559 generic.go:356] "Generic (PLEG): container finished" podID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerID="cbb263564417eac4faac1c82ce90a93c82881cabc20cbb8d39d8f9724bed5532" exitCode=0 Sep 08 02:03:30.672274 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:30.671877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" event={"ID":"a553c8db-1713-4476-8c5d-ba8a34c76345","Type":"ContainerDied","Data":"cbb263564417eac4faac1c82ce90a93c82881cabc20cbb8d39d8f9724bed5532"} Sep 08 02:03:31.840168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.840138 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:31.965183 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.965161 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle\") pod \"a553c8db-1713-4476-8c5d-ba8a34c76345\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " Sep 08 02:03:31.965316 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.965204 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bp2k\" (UniqueName: \"kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k\") pod \"a553c8db-1713-4476-8c5d-ba8a34c76345\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " Sep 08 02:03:31.965316 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.965230 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util\") pod \"a553c8db-1713-4476-8c5d-ba8a34c76345\" (UID: \"a553c8db-1713-4476-8c5d-ba8a34c76345\") " Sep 08 02:03:31.965951 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.965914 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle" (OuterVolumeSpecName: "bundle") pod "a553c8db-1713-4476-8c5d-ba8a34c76345" (UID: "a553c8db-1713-4476-8c5d-ba8a34c76345"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:03:31.967005 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.966983 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k" (OuterVolumeSpecName: "kube-api-access-2bp2k") pod "a553c8db-1713-4476-8c5d-ba8a34c76345" (UID: "a553c8db-1713-4476-8c5d-ba8a34c76345"). InnerVolumeSpecName "kube-api-access-2bp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:03:31.971016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:31.970996 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util" (OuterVolumeSpecName: "util") pod "a553c8db-1713-4476-8c5d-ba8a34c76345" (UID: "a553c8db-1713-4476-8c5d-ba8a34c76345"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:03:32.066781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.066745 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:32.066896 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.066784 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bp2k\" (UniqueName: \"kubernetes.io/projected/a553c8db-1713-4476-8c5d-ba8a34c76345-kube-api-access-2bp2k\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:32.066896 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.066801 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a553c8db-1713-4476-8c5d-ba8a34c76345-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:03:32.684665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.684634 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" Sep 08 02:03:32.684665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.684654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02bsnd" event={"ID":"a553c8db-1713-4476-8c5d-ba8a34c76345","Type":"ContainerDied","Data":"460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba"} Sep 08 02:03:32.684917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.684707 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460023a029e993738ca53488a730d0a77ce3c16a6e9f65a15085f7918dccbcba" Sep 08 02:03:32.686330 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:32.686277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" event={"ID":"577fd71c-6d67-4013-829a-055a9cddf509","Type":"ContainerStarted","Data":"2d810189e2853f94ec6c57c1c9df59097cd4c324f287c0f097014ca8341a8e27"} Sep 08 02:03:33.692323 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:33.692271 2559 generic.go:356] "Generic (PLEG): container finished" podID="577fd71c-6d67-4013-829a-055a9cddf509" containerID="2d810189e2853f94ec6c57c1c9df59097cd4c324f287c0f097014ca8341a8e27" exitCode=0 Sep 08 02:03:33.692789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:33.692360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" event={"ID":"577fd71c-6d67-4013-829a-055a9cddf509","Type":"ContainerDied","Data":"2d810189e2853f94ec6c57c1c9df59097cd4c324f287c0f097014ca8341a8e27"} Sep 08 02:03:34.699995 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:34.699961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" event={"ID":"577fd71c-6d67-4013-829a-055a9cddf509","Type":"ContainerStarted","Data":"94a80a00753392cd933c9a744f19b1e096f28536e8c47a38f6568b9f68f9aa27"} Sep 08 02:03:34.700391 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:34.700045 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:34.721226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:34.721180 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" podStartSLOduration=1.943037167 podStartE2EDuration="6.721166417s" totalImagesPullingTime="3.766601655s" totalInitContainerRuntime="1.011527595s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:34.72076188 +0000 UTC m=+295.898073952" watchObservedRunningTime="2026-09-08 02:03:34.721166417 +0000 UTC m=+295.898478488" Sep 08 02:03:39.254863 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:39.254829 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:03:39.267133 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:39.267103 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:03:39.274220 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:39.274189 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 02:03:45.707361 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:45.707326 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-dcbvk" Sep 08 02:03:47.694695 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.694660 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr"] Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.694977 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="util" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.694990 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="util" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.695001 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="extract" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.695006 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="extract" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.695016 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="pull" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.695022 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="pull" Sep 08 02:03:47.695066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.695070 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a553c8db-1713-4476-8c5d-ba8a34c76345" containerName="extract" Sep 08 02:03:47.697938 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.697922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.706263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.706245 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 02:03:47.707411 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.707392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vkdjb\"" Sep 08 02:03:47.707522 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.707429 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 02:03:47.786521 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.786485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5xl\" (UniqueName: \"kubernetes.io/projected/edadf9cc-8d04-4944-ad81-0e29df02b766-kube-api-access-td5xl\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.786685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.786534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadf9cc-8d04-4944-ad81-0e29df02b766-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.786685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.786609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edadf9cc-8d04-4944-ad81-0e29df02b766-tmp\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.810011 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.809983 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr"] Sep 08 02:03:47.887458 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.887428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5xl\" (UniqueName: \"kubernetes.io/projected/edadf9cc-8d04-4944-ad81-0e29df02b766-kube-api-access-td5xl\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.887592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.887474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadf9cc-8d04-4944-ad81-0e29df02b766-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.887592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.887508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edadf9cc-8d04-4944-ad81-0e29df02b766-tmp\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.889690 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.889661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edadf9cc-8d04-4944-ad81-0e29df02b766-tmp\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.889896 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.889877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edadf9cc-8d04-4944-ad81-0e29df02b766-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:47.928117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:47.928085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5xl\" (UniqueName: \"kubernetes.io/projected/edadf9cc-8d04-4944-ad81-0e29df02b766-kube-api-access-td5xl\") pod \"kube-auth-proxy-9c6fc765d-9pnbr\" (UID: \"edadf9cc-8d04-4944-ad81-0e29df02b766\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:48.007524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:48.007460 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" Sep 08 02:03:48.278801 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:48.278728 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr"] Sep 08 02:03:48.282171 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:03:48.282142 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedadf9cc_8d04_4944_ad81_0e29df02b766.slice/crio-2bca464ad2dfc055645c0c188247c278524f50d68bf9dd77d4ac9fb9c30bcdf5 WatchSource:0}: Error finding container 2bca464ad2dfc055645c0c188247c278524f50d68bf9dd77d4ac9fb9c30bcdf5: Status 404 returned error can't find the container with id 2bca464ad2dfc055645c0c188247c278524f50d68bf9dd77d4ac9fb9c30bcdf5 Sep 08 02:03:48.283943 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:48.283922 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 02:03:48.761061 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:48.761027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" event={"ID":"edadf9cc-8d04-4944-ad81-0e29df02b766","Type":"ContainerStarted","Data":"2bca464ad2dfc055645c0c188247c278524f50d68bf9dd77d4ac9fb9c30bcdf5"} Sep 08 02:03:49.302353 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.302279 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:03:49.305803 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.305716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.310697 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.310674 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 02:03:49.312031 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.311947 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 02:03:49.312031 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.311960 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 02:03:49.312031 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.312026 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 02:03:49.312244 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.312102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-drqj9\"" Sep 08 02:03:49.323161 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.323141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:03:49.401263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdz6\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.401602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.401444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.502667 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdz6\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.502841 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.502841 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.502841 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.502841 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.503067 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.503067 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.502912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.503341 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.503315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.506045 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.506000 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.506045 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.506018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.506045 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.506040 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.506342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.506301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.511238 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.511207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.512038 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.512015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdz6\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6\") pod \"istiod-openshift-gateway-7b7995d5cd-hh9rj\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.625647 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.625561 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:49.917576 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:49.917542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:03:51.132128 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:03:51.132095 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0465a958_1c45_4847_8b0f_05cc2b14f1ac.slice/crio-1b6cd2cd57bb00cb6910a4f1a677c4c99f26bbef8ecc73c71ad19079da5ab5b8 WatchSource:0}: Error finding container 1b6cd2cd57bb00cb6910a4f1a677c4c99f26bbef8ecc73c71ad19079da5ab5b8: Status 404 returned error can't find the container with id 1b6cd2cd57bb00cb6910a4f1a677c4c99f26bbef8ecc73c71ad19079da5ab5b8 Sep 08 02:03:51.778133 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:51.778092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" event={"ID":"edadf9cc-8d04-4944-ad81-0e29df02b766","Type":"ContainerStarted","Data":"5396b9e9c9476a96c0186eed38dbd5ef0e135de0eeea29d9854120128e63133d"} Sep 08 02:03:51.779308 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:51.779262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" event={"ID":"0465a958-1c45-4847-8b0f-05cc2b14f1ac","Type":"ContainerStarted","Data":"1b6cd2cd57bb00cb6910a4f1a677c4c99f26bbef8ecc73c71ad19079da5ab5b8"} Sep 08 02:03:51.796437 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:51.796386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-9pnbr" podStartSLOduration=1.8956268490000001 podStartE2EDuration="4.796371666s" totalImagesPullingTime="2.900744817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:51.794809046 +0000 UTC m=+312.972121122" watchObservedRunningTime="2026-09-08 02:03:51.796371666 +0000 UTC m=+312.973683738" Sep 08 02:03:54.242524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:54.242487 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:03:54.242857 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:54.242555 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:03:54.800049 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:54.799968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" event={"ID":"0465a958-1c45-4847-8b0f-05cc2b14f1ac","Type":"ContainerStarted","Data":"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d"} Sep 08 02:03:54.800283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:54.800142 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:03:54.825386 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:54.825323 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" podStartSLOduration=2.717628307 podStartE2EDuration="5.825303411s" totalImagesPullingTime="3.107675104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:54.822256987 +0000 UTC m=+315.999569058" watchObservedRunningTime="2026-09-08 02:03:54.825303411 +0000 UTC m=+316.002615481" Sep 08 02:03:55.806385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:03:55.806354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:04:09.665161 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.665123 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr"] Sep 08 02:04:09.675207 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.675179 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r"] Sep 08 02:04:09.675396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.675328 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.678520 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.678496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.678997 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.678976 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-snxhw\"" Sep 08 02:04:09.680397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.680378 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 08 02:04:09.683023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.682996 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8sb5j\"" Sep 08 02:04:09.683351 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.683336 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 08 02:04:09.695044 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.695017 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr"] Sep 08 02:04:09.767732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.767698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r"] Sep 08 02:04:09.771283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.771254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e36c67-3234-40df-bc1d-4e1075a72ac5-cert\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.771441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.771325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfvr\" (UniqueName: \"kubernetes.io/projected/e863c99e-9508-412b-8687-65b83172f716-kube-api-access-mgfvr\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.771441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.771359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkj5l\" (UniqueName: \"kubernetes.io/projected/f5e36c67-3234-40df-bc1d-4e1075a72ac5-kube-api-access-hkj5l\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.771441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.771410 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.872645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.872611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e36c67-3234-40df-bc1d-4e1075a72ac5-cert\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.872830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.872662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfvr\" (UniqueName: \"kubernetes.io/projected/e863c99e-9508-412b-8687-65b83172f716-kube-api-access-mgfvr\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.872830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.872680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkj5l\" (UniqueName: \"kubernetes.io/projected/f5e36c67-3234-40df-bc1d-4e1075a72ac5-kube-api-access-hkj5l\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.872830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.872711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.872830 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:04:09.872800 2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 08 02:04:09.873066 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:04:09.872864 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert podName:e863c99e-9508-412b-8687-65b83172f716 nodeName:}" failed. No retries permitted until 2026-09-08 02:04:10.372848434 +0000 UTC m=+331.550160483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert") pod "llmisvc-controller-manager-777c6bd7d5-8nd6r" (UID: "e863c99e-9508-412b-8687-65b83172f716") : secret "llmisvc-webhook-server-cert" not found Sep 08 02:04:09.875303 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.875267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e36c67-3234-40df-bc1d-4e1075a72ac5-cert\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.887114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.887088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfvr\" (UniqueName: \"kubernetes.io/projected/e863c99e-9508-412b-8687-65b83172f716-kube-api-access-mgfvr\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:09.887262 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.887137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkj5l\" (UniqueName: \"kubernetes.io/projected/f5e36c67-3234-40df-bc1d-4e1075a72ac5-kube-api-access-hkj5l\") pod \"kserve-controller-manager-d6f4b65b8-cs8mr\" (UID: \"f5e36c67-3234-40df-bc1d-4e1075a72ac5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:09.986404 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:09.986322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:10.259065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.259038 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr"] Sep 08 02:04:10.260881 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:04:10.260859 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e36c67_3234_40df_bc1d_4e1075a72ac5.slice/crio-7cc17300df7b18083046ffdafeb89e9e4bebad2fd5580c9cf28d0030e7a1062f WatchSource:0}: Error finding container 7cc17300df7b18083046ffdafeb89e9e4bebad2fd5580c9cf28d0030e7a1062f: Status 404 returned error can't find the container with id 7cc17300df7b18083046ffdafeb89e9e4bebad2fd5580c9cf28d0030e7a1062f Sep 08 02:04:10.377980 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.377933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:10.380329 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.380282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e863c99e-9508-412b-8687-65b83172f716-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nd6r\" (UID: \"e863c99e-9508-412b-8687-65b83172f716\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:10.593362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.593245 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:10.779230 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.779184 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:04:10.779733 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.779491 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerName="discovery" containerID="cri-o://4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d" gracePeriod=30 Sep 08 02:04:10.805116 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.805079 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-hh9rj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:8080/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" start-of-body= Sep 08 02:04:10.805272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.805137 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.33:8080/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 08 02:04:10.868488 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.868452 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r"] Sep 08 02:04:10.870415 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:04:10.870388 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode863c99e_9508_412b_8687_65b83172f716.slice/crio-294a076f5a68f5cfb23bb859059d6a348354d9451556ccf092e9f3ec8cea8f68 WatchSource:0}: Error finding container 294a076f5a68f5cfb23bb859059d6a348354d9451556ccf092e9f3ec8cea8f68: Status 404 returned error can't find the container with id 294a076f5a68f5cfb23bb859059d6a348354d9451556ccf092e9f3ec8cea8f68 Sep 08 02:04:10.875740 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:10.875715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" event={"ID":"f5e36c67-3234-40df-bc1d-4e1075a72ac5","Type":"ContainerStarted","Data":"7cc17300df7b18083046ffdafeb89e9e4bebad2fd5580c9cf28d0030e7a1062f"} Sep 08 02:04:11.046742 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.046711 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:04:11.185737 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185701 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctdz6\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.185930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185774 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.185930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.185930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185839 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.185930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185877 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.185930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185904 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.186175 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.185934 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig\") pod \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\" (UID: \"0465a958-1c45-4847-8b0f-05cc2b14f1ac\") " Sep 08 02:04:11.186616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.186578 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:04:11.188992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.188798 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6" (OuterVolumeSpecName: "kube-api-access-ctdz6") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "kube-api-access-ctdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:04:11.188992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.188811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs" (OuterVolumeSpecName: "local-certs") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:04:11.188992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.188874 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:04:11.189263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.189202 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:04:11.189433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.189405 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts" (OuterVolumeSpecName: "cacerts") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:04:11.189678 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.189656 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "0465a958-1c45-4847-8b0f-05cc2b14f1ac" (UID: "0465a958-1c45-4847-8b0f-05cc2b14f1ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:04:11.287542 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287502 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdz6\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-kube-api-access-ctdz6\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287542 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287535 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-token\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287542 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287549 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-cacerts\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287559 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0465a958-1c45-4847-8b0f-05cc2b14f1ac-local-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287572 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-ca-configmap\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287583 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-csr-dns-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.287777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.287596 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0465a958-1c45-4847-8b0f-05cc2b14f1ac-istio-kubeconfig\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:04:11.881085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.881052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" event={"ID":"e863c99e-9508-412b-8687-65b83172f716","Type":"ContainerStarted","Data":"294a076f5a68f5cfb23bb859059d6a348354d9451556ccf092e9f3ec8cea8f68"} Sep 08 02:04:11.882448 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.882418 2559 generic.go:356] "Generic (PLEG): container finished" podID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerID="4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d" exitCode=0 Sep 08 02:04:11.882578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.882469 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" Sep 08 02:04:11.882578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.882482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" event={"ID":"0465a958-1c45-4847-8b0f-05cc2b14f1ac","Type":"ContainerDied","Data":"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d"} Sep 08 02:04:11.882578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.882521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj" event={"ID":"0465a958-1c45-4847-8b0f-05cc2b14f1ac","Type":"ContainerDied","Data":"1b6cd2cd57bb00cb6910a4f1a677c4c99f26bbef8ecc73c71ad19079da5ab5b8"} Sep 08 02:04:11.882578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.882543 2559 scope.go:117] "RemoveContainer" containerID="4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d" Sep 08 02:04:11.902432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.902394 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:04:11.907444 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.907416 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hh9rj"] Sep 08 02:04:11.970783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.970746 2559 scope.go:117] "RemoveContainer" containerID="4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d" Sep 08 02:04:11.971196 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:04:11.971168 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d\": container with ID starting with 4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d not found: ID does not exist" containerID="4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d" Sep 08 02:04:11.971308 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:11.971211 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d"} err="failed to get container status \"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d\": rpc error: code = NotFound desc = could not find container \"4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d\": container with ID starting with 4e388e491725be6b06a33662a95073cd682630c71fb096ef8726fae85edbee6d not found: ID does not exist" Sep 08 02:04:12.892522 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:12.892483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" event={"ID":"f5e36c67-3234-40df-bc1d-4e1075a72ac5","Type":"ContainerStarted","Data":"c476b79bce7e9888d0fb3aca9f8c62107fc560a05b9831dc7931a6321a4325d1"} Sep 08 02:04:12.893105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:12.892616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:12.920574 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:12.920523 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" podStartSLOduration=1.401192408 podStartE2EDuration="3.920501864s" totalImagesPullingTime="2.519309456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:04:12.918665276 +0000 UTC m=+334.095977347" watchObservedRunningTime="2026-09-08 02:04:12.920501864 +0000 UTC m=+334.097813936" Sep 08 02:04:13.319448 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:13.319416 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" path="/var/lib/kubelet/pods/0465a958-1c45-4847-8b0f-05cc2b14f1ac/volumes" Sep 08 02:04:13.898789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:13.898695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" event={"ID":"e863c99e-9508-412b-8687-65b83172f716","Type":"ContainerStarted","Data":"995e8973bd529caa967b4ea41154c355e14fee6769ed16ab534cf366d0c750a0"} Sep 08 02:04:13.899188 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:13.898933 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:13.915393 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:13.915351 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" podStartSLOduration=2.153038537 podStartE2EDuration="4.915337886s" totalImagesPullingTime="2.762299349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:04:13.914913115 +0000 UTC m=+335.092225186" watchObservedRunningTime="2026-09-08 02:04:13.915337886 +0000 UTC m=+335.092649953" Sep 08 02:04:43.904392 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:43.904358 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cs8mr" Sep 08 02:04:44.907126 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:44.907094 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nd6r" Sep 08 02:04:51.377956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.377921 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b7975f6cb-hnzfc"] Sep 08 02:04:51.378346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.378232 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerName="discovery" Sep 08 02:04:51.378346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.378242 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerName="discovery" Sep 08 02:04:51.378346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.378312 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0465a958-1c45-4847-8b0f-05cc2b14f1ac" containerName="discovery" Sep 08 02:04:51.385013 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.384992 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.391412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.391386 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7975f6cb-hnzfc"] Sep 08 02:04:51.409682 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-oauth-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.409855 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-trusted-ca-bundle\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.409855 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z447z\" (UniqueName: \"kubernetes.io/projected/ad9a1638-4452-4cd0-80d9-83365238be0a-kube-api-access-z447z\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.409855 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.409855 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-service-ca\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.410019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409866 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-oauth-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.410019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.409903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-console-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-trusted-ca-bundle\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z447z\" (UniqueName: \"kubernetes.io/projected/ad9a1638-4452-4cd0-80d9-83365238be0a-kube-api-access-z447z\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-service-ca\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-oauth-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-console-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.511633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.511618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-oauth-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.512389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.512358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-service-ca\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.512538 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.512416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-trusted-ca-bundle\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.512538 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.512451 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-oauth-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.512538 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.512484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad9a1638-4452-4cd0-80d9-83365238be0a-console-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.514119 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.514098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-serving-cert\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.514412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.514394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad9a1638-4452-4cd0-80d9-83365238be0a-console-oauth-config\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.518760 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.518740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z447z\" (UniqueName: \"kubernetes.io/projected/ad9a1638-4452-4cd0-80d9-83365238be0a-kube-api-access-z447z\") pod \"console-5b7975f6cb-hnzfc\" (UID: \"ad9a1638-4452-4cd0-80d9-83365238be0a\") " pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.695930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.695894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:04:51.969194 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:51.969169 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7975f6cb-hnzfc"] Sep 08 02:04:51.971479 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:04:51.971451 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9a1638_4452_4cd0_80d9_83365238be0a.slice/crio-d900c6d8848277430ac1cddf43703bef861112e21ef62699321946b3376f9351 WatchSource:0}: Error finding container d900c6d8848277430ac1cddf43703bef861112e21ef62699321946b3376f9351: Status 404 returned error can't find the container with id d900c6d8848277430ac1cddf43703bef861112e21ef62699321946b3376f9351 Sep 08 02:04:52.078007 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:52.077968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7975f6cb-hnzfc" event={"ID":"ad9a1638-4452-4cd0-80d9-83365238be0a","Type":"ContainerStarted","Data":"3dd45ee37d54e754841247688b63bcb531f09b95809faafd4ef3ca61c6621c57"} Sep 08 02:04:52.078155 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:52.078030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7975f6cb-hnzfc" event={"ID":"ad9a1638-4452-4cd0-80d9-83365238be0a","Type":"ContainerStarted","Data":"d900c6d8848277430ac1cddf43703bef861112e21ef62699321946b3376f9351"} Sep 08 02:04:52.094754 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:04:52.094706 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b7975f6cb-hnzfc" podStartSLOduration=1.094689491 podStartE2EDuration="1.094689491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:04:52.093355534 +0000 UTC m=+373.270667604" watchObservedRunningTime="2026-09-08 02:04:52.094689491 +0000 UTC m=+373.272001562" Sep 08 02:05:01.696931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:01.696890 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:05:01.697612 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:01.696956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:05:01.701717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:01.701694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:05:02.126816 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.126740 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b7975f6cb-hnzfc" Sep 08 02:05:02.171616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.171583 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:05:02.832562 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.832520 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc"] Sep 08 02:05:02.836048 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.836031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:02.838120 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.838097 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 02:05:02.838926 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.838909 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 02:05:02.838986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.838944 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vqx5j\"" Sep 08 02:05:02.842524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.842503 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc"] Sep 08 02:05:02.903272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.903245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qks86\" (UniqueName: \"kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:02.903433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.903388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:02.903433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.903414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:02.932013 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.931987 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn"] Sep 08 02:05:02.935482 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.935467 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:02.941713 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:02.941691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn"] Sep 08 02:05:03.004367 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004326 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks86\" (UniqueName: \"kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.004552 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.004552 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.004552 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.004700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.004700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxvf\" (UniqueName: \"kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.004873 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.004908 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.004880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.013681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.013663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qks86\" (UniqueName: \"kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.027235 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.027208 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2"] Sep 08 02:05:03.030745 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.030731 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.038365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.038347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2"] Sep 08 02:05:03.105719 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.105719 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.105914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.105914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.105914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxvf\" (UniqueName: \"kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.105914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.105842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd7w\" (UniqueName: \"kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.106119 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.106013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.106119 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.106098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.115193 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.115162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxvf\" (UniqueName: \"kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.131271 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.131236 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f"] Sep 08 02:05:03.134731 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.134715 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.142975 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.142950 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f"] Sep 08 02:05:03.146815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.146794 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:03.206511 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.206511 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.206703 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd7w\" (UniqueName: \"kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.206703 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206618 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2nj\" (UniqueName: \"kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.206807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.206807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.206785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.207394 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.207316 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.207394 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.207359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.217966 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.217912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd7w\" (UniqueName: \"kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.244469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.244439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:03.308023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.307661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.308023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.307749 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.308023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.307843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2nj\" (UniqueName: \"kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.308347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.308214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.308347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.308315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.316656 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.316602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2nj\" (UniqueName: \"kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.340101 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.340064 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:03.423315 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.423278 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc"] Sep 08 02:05:03.425127 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:03.425095 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab612bf5_3d5d_46a5_99c3_1acb0a0f1239.slice/crio-f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0 WatchSource:0}: Error finding container f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0: Status 404 returned error can't find the container with id f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0 Sep 08 02:05:03.444594 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.444574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:03.522894 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.522871 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn"] Sep 08 02:05:03.524616 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:03.524589 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae933c54_841f_44e8_abaf_af07fb635183.slice/crio-732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864 WatchSource:0}: Error finding container 732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864: Status 404 returned error can't find the container with id 732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864 Sep 08 02:05:03.615617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.615593 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2"] Sep 08 02:05:03.617758 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:03.617733 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc5bd33_7f2d_4865_ab4d_2efd62b50d67.slice/crio-7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a WatchSource:0}: Error finding container 7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a: Status 404 returned error can't find the container with id 7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a Sep 08 02:05:03.717114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:03.717091 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f"] Sep 08 02:05:03.750035 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:03.750007 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fa5548_aa32_4b94_a57d_f18ffa3016fd.slice/crio-81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37 WatchSource:0}: Error finding container 81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37: Status 404 returned error can't find the container with id 81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37 Sep 08 02:05:04.134534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.134441 2559 generic.go:356] "Generic (PLEG): container finished" podID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerID="fd23ab6131f5e0e04c4365efca8f0b50e2928fd627375b940c70ff4c9036116a" exitCode=0 Sep 08 02:05:04.134534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.134512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" event={"ID":"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67","Type":"ContainerDied","Data":"fd23ab6131f5e0e04c4365efca8f0b50e2928fd627375b940c70ff4c9036116a"} Sep 08 02:05:04.135009 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.134581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" event={"ID":"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67","Type":"ContainerStarted","Data":"7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a"} Sep 08 02:05:04.136167 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.136144 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae933c54-841f-44e8-abaf-af07fb635183" containerID="031e6f4574d7a5e0701204df9a41ef7e0466cbea10b68caf0704e647ac657204" exitCode=0 Sep 08 02:05:04.136241 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.136221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" event={"ID":"ae933c54-841f-44e8-abaf-af07fb635183","Type":"ContainerDied","Data":"031e6f4574d7a5e0701204df9a41ef7e0466cbea10b68caf0704e647ac657204"} Sep 08 02:05:04.136312 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.136257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" event={"ID":"ae933c54-841f-44e8-abaf-af07fb635183","Type":"ContainerStarted","Data":"732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864"} Sep 08 02:05:04.137892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.137870 2559 generic.go:356] "Generic (PLEG): container finished" podID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerID="1de9ab3454d71ddb1c0331fd392a6fbadc676a74b737ca96692e70bdc05d4992" exitCode=0 Sep 08 02:05:04.137965 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.137941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" event={"ID":"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239","Type":"ContainerDied","Data":"1de9ab3454d71ddb1c0331fd392a6fbadc676a74b737ca96692e70bdc05d4992"} Sep 08 02:05:04.138026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.137973 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" event={"ID":"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239","Type":"ContainerStarted","Data":"f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0"} Sep 08 02:05:04.139703 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.139681 2559 generic.go:356] "Generic (PLEG): container finished" podID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerID="76c0bf4c51560ad0bcc2553103b6e48dcc30c8f6b0f8bd8e29f4147e6c9f1d58" exitCode=0 Sep 08 02:05:04.139789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.139753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" event={"ID":"03fa5548-aa32-4b94-a57d-f18ffa3016fd","Type":"ContainerDied","Data":"76c0bf4c51560ad0bcc2553103b6e48dcc30c8f6b0f8bd8e29f4147e6c9f1d58"} Sep 08 02:05:04.139789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:04.139784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" event={"ID":"03fa5548-aa32-4b94-a57d-f18ffa3016fd","Type":"ContainerStarted","Data":"81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37"} Sep 08 02:05:06.152036 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.152002 2559 generic.go:356] "Generic (PLEG): container finished" podID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerID="3b4a8184ad8227f78d3034ddb8d10d25cf6b78fe25b70c285e74a351611a8930" exitCode=0 Sep 08 02:05:06.152506 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.152074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" event={"ID":"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67","Type":"ContainerDied","Data":"3b4a8184ad8227f78d3034ddb8d10d25cf6b78fe25b70c285e74a351611a8930"} Sep 08 02:05:06.153948 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.153927 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae933c54-841f-44e8-abaf-af07fb635183" containerID="157ecdebcb77e25b569936319918b986518792aaf9ab1b437850dd50b683e822" exitCode=0 Sep 08 02:05:06.154042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.153987 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" event={"ID":"ae933c54-841f-44e8-abaf-af07fb635183","Type":"ContainerDied","Data":"157ecdebcb77e25b569936319918b986518792aaf9ab1b437850dd50b683e822"} Sep 08 02:05:06.155915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.155896 2559 generic.go:356] "Generic (PLEG): container finished" podID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerID="5a6aeb0673bc82d15ca9ca693197bbed9bc43d4d79acfa13bb7a6ac6c62c3084" exitCode=0 Sep 08 02:05:06.155999 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.155982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" event={"ID":"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239","Type":"ContainerDied","Data":"5a6aeb0673bc82d15ca9ca693197bbed9bc43d4d79acfa13bb7a6ac6c62c3084"} Sep 08 02:05:06.158118 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.158097 2559 generic.go:356] "Generic (PLEG): container finished" podID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerID="dfc54b2cca83ba865d494d52c853ee859596391d2bbf1529b8cdeabf21c42cc7" exitCode=0 Sep 08 02:05:06.158213 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:06.158146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" event={"ID":"03fa5548-aa32-4b94-a57d-f18ffa3016fd","Type":"ContainerDied","Data":"dfc54b2cca83ba865d494d52c853ee859596391d2bbf1529b8cdeabf21c42cc7"} Sep 08 02:05:07.167030 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.166992 2559 generic.go:356] "Generic (PLEG): container finished" podID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerID="86aa2efd5ef86eb9dbabd1cd62118002cabb301bcf5ba721bd3b571e132d07ec" exitCode=0 Sep 08 02:05:07.167479 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.167073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" event={"ID":"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67","Type":"ContainerDied","Data":"86aa2efd5ef86eb9dbabd1cd62118002cabb301bcf5ba721bd3b571e132d07ec"} Sep 08 02:05:07.169306 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.169271 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae933c54-841f-44e8-abaf-af07fb635183" containerID="c10a0e3218d1487f9f0464b578fab9dc446b2f79668ecfb7bb3f0ccd8defa3d8" exitCode=0 Sep 08 02:05:07.169404 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.169318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" event={"ID":"ae933c54-841f-44e8-abaf-af07fb635183","Type":"ContainerDied","Data":"c10a0e3218d1487f9f0464b578fab9dc446b2f79668ecfb7bb3f0ccd8defa3d8"} Sep 08 02:05:07.171443 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.171420 2559 generic.go:356] "Generic (PLEG): container finished" podID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerID="817729bdb3effb16c8f7aa7f27a85141e328f0ed72b52959572968d09a5e5b82" exitCode=0 Sep 08 02:05:07.171552 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.171480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" event={"ID":"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239","Type":"ContainerDied","Data":"817729bdb3effb16c8f7aa7f27a85141e328f0ed72b52959572968d09a5e5b82"} Sep 08 02:05:07.173817 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.173797 2559 generic.go:356] "Generic (PLEG): container finished" podID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerID="b0f63957f4212ef4207abeb6f1287b063eb2681544c3cb11ce0644434455e0fb" exitCode=0 Sep 08 02:05:07.173904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:07.173855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" event={"ID":"03fa5548-aa32-4b94-a57d-f18ffa3016fd","Type":"ContainerDied","Data":"b0f63957f4212ef4207abeb6f1287b063eb2681544c3cb11ce0644434455e0fb"} Sep 08 02:05:08.321169 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.321137 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:08.350636 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.350605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg2nj\" (UniqueName: \"kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj\") pod \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " Sep 08 02:05:08.350751 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.350669 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util\") pod \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " Sep 08 02:05:08.350751 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.350713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle\") pod \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\" (UID: \"03fa5548-aa32-4b94-a57d-f18ffa3016fd\") " Sep 08 02:05:08.351995 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.351969 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle" (OuterVolumeSpecName: "bundle") pod "03fa5548-aa32-4b94-a57d-f18ffa3016fd" (UID: "03fa5548-aa32-4b94-a57d-f18ffa3016fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.358068 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.358042 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj" (OuterVolumeSpecName: "kube-api-access-pg2nj") pod "03fa5548-aa32-4b94-a57d-f18ffa3016fd" (UID: "03fa5548-aa32-4b94-a57d-f18ffa3016fd"). InnerVolumeSpecName "kube-api-access-pg2nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:08.358650 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.358625 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util" (OuterVolumeSpecName: "util") pod "03fa5548-aa32-4b94-a57d-f18ffa3016fd" (UID: "03fa5548-aa32-4b94-a57d-f18ffa3016fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.384578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.384550 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:08.386690 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.386668 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:08.390917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.390901 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:08.451800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle\") pod \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " Sep 08 02:05:08.451800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451749 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qks86\" (UniqueName: \"kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86\") pod \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " Sep 08 02:05:08.451800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451785 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util\") pod \"ae933c54-841f-44e8-abaf-af07fb635183\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " Sep 08 02:05:08.451800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451801 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle\") pod \"ae933c54-841f-44e8-abaf-af07fb635183\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " Sep 08 02:05:08.452121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451823 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsxvf\" (UniqueName: \"kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf\") pod \"ae933c54-841f-44e8-abaf-af07fb635183\" (UID: \"ae933c54-841f-44e8-abaf-af07fb635183\") " Sep 08 02:05:08.452121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util\") pod \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " Sep 08 02:05:08.452121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451900 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util\") pod \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " Sep 08 02:05:08.452121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle\") pod \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\" (UID: \"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239\") " Sep 08 02:05:08.452121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.451950 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sd7w\" (UniqueName: \"kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w\") pod \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\" (UID: \"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67\") " Sep 08 02:05:08.452380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452216 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2nj\" (UniqueName: \"kubernetes.io/projected/03fa5548-aa32-4b94-a57d-f18ffa3016fd-kube-api-access-pg2nj\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.452380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452235 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.452380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452246 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03fa5548-aa32-4b94-a57d-f18ffa3016fd-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.452784 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452436 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle" (OuterVolumeSpecName: "bundle") pod "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" (UID: "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.452784 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452439 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle" (OuterVolumeSpecName: "bundle") pod "ae933c54-841f-44e8-abaf-af07fb635183" (UID: "ae933c54-841f-44e8-abaf-af07fb635183"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.452963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.452929 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle" (OuterVolumeSpecName: "bundle") pod "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" (UID: "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.454620 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.454596 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf" (OuterVolumeSpecName: "kube-api-access-bsxvf") pod "ae933c54-841f-44e8-abaf-af07fb635183" (UID: "ae933c54-841f-44e8-abaf-af07fb635183"). InnerVolumeSpecName "kube-api-access-bsxvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:08.454729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.454626 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w" (OuterVolumeSpecName: "kube-api-access-5sd7w") pod "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" (UID: "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67"). InnerVolumeSpecName "kube-api-access-5sd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:08.455172 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.455157 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86" (OuterVolumeSpecName: "kube-api-access-qks86") pod "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" (UID: "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239"). InnerVolumeSpecName "kube-api-access-qks86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:08.458414 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.458380 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util" (OuterVolumeSpecName: "util") pod "ae933c54-841f-44e8-abaf-af07fb635183" (UID: "ae933c54-841f-44e8-abaf-af07fb635183"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.461533 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.461503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util" (OuterVolumeSpecName: "util") pod "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" (UID: "5cc5bd33-7f2d-4865-ab4d-2efd62b50d67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.461815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.461797 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util" (OuterVolumeSpecName: "util") pod "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" (UID: "ab612bf5-3d5d-46a5-99c3-1acb0a0f1239"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:08.552866 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552828 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxvf\" (UniqueName: \"kubernetes.io/projected/ae933c54-841f-44e8-abaf-af07fb635183-kube-api-access-bsxvf\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.552866 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552862 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.552866 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552871 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552879 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552887 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd7w\" (UniqueName: \"kubernetes.io/projected/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-kube-api-access-5sd7w\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552896 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc5bd33-7f2d-4865-ab4d-2efd62b50d67-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552905 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qks86\" (UniqueName: \"kubernetes.io/projected/ab612bf5-3d5d-46a5-99c3-1acb0a0f1239-kube-api-access-qks86\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552912 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-util\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:08.553091 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:08.552920 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae933c54-841f-44e8-abaf-af07fb635183-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:09.185773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.185746 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" Sep 08 02:05:09.185948 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.185745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4q8g2" event={"ID":"5cc5bd33-7f2d-4865-ab4d-2efd62b50d67","Type":"ContainerDied","Data":"7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a"} Sep 08 02:05:09.185948 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.185883 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acc0852b2a0b51533a2c87214b67f597b210868f76923ae254ebd98bd8e253a" Sep 08 02:05:09.187712 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.187690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" event={"ID":"ae933c54-841f-44e8-abaf-af07fb635183","Type":"ContainerDied","Data":"732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864"} Sep 08 02:05:09.187712 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.187701 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsv9bn" Sep 08 02:05:09.187901 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.187726 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732b366b54f6ef4743eb3e900732463db34babf4c18e47f94491cf5194dab864" Sep 08 02:05:09.189572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.189556 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" Sep 08 02:05:09.189665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.189585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rn5rc" event={"ID":"ab612bf5-3d5d-46a5-99c3-1acb0a0f1239","Type":"ContainerDied","Data":"f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0"} Sep 08 02:05:09.189665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.189621 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19fe9f6c5f3efcb37ada5d3b3e91743c5816d3817e4f7e6253f3fd4a96076f0" Sep 08 02:05:09.191524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.191506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" event={"ID":"03fa5548-aa32-4b94-a57d-f18ffa3016fd","Type":"ContainerDied","Data":"81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37"} Sep 08 02:05:09.191609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.191533 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81fe5244b712f5a3332740cfafe2afeac32f101330035136d3b01919a6007c37" Sep 08 02:05:09.191609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:09.191541 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053l5j8f" Sep 08 02:05:14.414135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414102 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7"] Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414443 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="pull" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414454 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="pull" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414463 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414473 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414483 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414489 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414501 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414506 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414512 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414516 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="util" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414521 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="extract" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414526 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="extract" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414532 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="pull" Sep 08 02:05:14.414535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414537 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="pull" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414546 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414554 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414564 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="pull" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414569 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="pull" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414575 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414580 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414586 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414591 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414598 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="pull" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414603 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="pull" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414659 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc5bd33-7f2d-4865-ab4d-2efd62b50d67" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414668 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab612bf5-3d5d-46a5-99c3-1acb0a0f1239" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414675 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fa5548-aa32-4b94-a57d-f18ffa3016fd" containerName="extract" Sep 08 02:05:14.414931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.414682 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae933c54-841f-44e8-abaf-af07fb635183" containerName="extract" Sep 08 02:05:14.418971 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.418954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:14.421350 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.421327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 02:05:14.421480 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.421327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 02:05:14.422202 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.422172 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-l7h9p\"" Sep 08 02:05:14.428182 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.428162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7"] Sep 08 02:05:14.502951 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.502932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8clm\" (UniqueName: \"kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm\") pod \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" (UID: \"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:14.603277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.603251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8clm\" (UniqueName: \"kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm\") pod \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" (UID: \"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:14.612436 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.612411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8clm\" (UniqueName: \"kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm\") pod \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" (UID: \"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:14.729075 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:14.728999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:15.007966 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:15.007943 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7"] Sep 08 02:05:15.009716 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:15.009690 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da2d30f_47e7_4ef0_ac21_f1b82ab4c0bb.slice/crio-92c8d701e3cefc989d19121cbce3e49aae9eb408ba795ff11ef7891cfb3ca081 WatchSource:0}: Error finding container 92c8d701e3cefc989d19121cbce3e49aae9eb408ba795ff11ef7891cfb3ca081: Status 404 returned error can't find the container with id 92c8d701e3cefc989d19121cbce3e49aae9eb408ba795ff11ef7891cfb3ca081 Sep 08 02:05:15.222679 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:15.222645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" event={"ID":"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb","Type":"ContainerStarted","Data":"92c8d701e3cefc989d19121cbce3e49aae9eb408ba795ff11ef7891cfb3ca081"} Sep 08 02:05:16.528959 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.528925 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq"] Sep 08 02:05:16.532649 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.532629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:16.535081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.535061 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 02:05:16.535189 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.535163 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zscbs\"" Sep 08 02:05:16.547491 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.547466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq"] Sep 08 02:05:16.619889 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.619849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwk6s\" (UniqueName: \"kubernetes.io/projected/77ebc803-9570-4ade-9655-2a0bb00d86cc-kube-api-access-zwk6s\") pod \"dns-operator-controller-manager-5985564776-gg6nq\" (UID: \"77ebc803-9570-4ade-9655-2a0bb00d86cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:16.720645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.720574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwk6s\" (UniqueName: \"kubernetes.io/projected/77ebc803-9570-4ade-9655-2a0bb00d86cc-kube-api-access-zwk6s\") pod \"dns-operator-controller-manager-5985564776-gg6nq\" (UID: \"77ebc803-9570-4ade-9655-2a0bb00d86cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:16.735597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.735566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwk6s\" (UniqueName: \"kubernetes.io/projected/77ebc803-9570-4ade-9655-2a0bb00d86cc-kube-api-access-zwk6s\") pod \"dns-operator-controller-manager-5985564776-gg6nq\" (UID: \"77ebc803-9570-4ade-9655-2a0bb00d86cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:16.846233 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:16.846194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:17.932369 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:17.932344 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq"] Sep 08 02:05:17.934112 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:17.934086 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ebc803_9570_4ade_9655_2a0bb00d86cc.slice/crio-c41c78becf18bfe0599d54d02c5318d317ad8aa0fba100eab1f29b292ee585c5 WatchSource:0}: Error finding container c41c78becf18bfe0599d54d02c5318d317ad8aa0fba100eab1f29b292ee585c5: Status 404 returned error can't find the container with id c41c78becf18bfe0599d54d02c5318d317ad8aa0fba100eab1f29b292ee585c5 Sep 08 02:05:18.244669 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:18.244579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" event={"ID":"77ebc803-9570-4ade-9655-2a0bb00d86cc","Type":"ContainerStarted","Data":"c41c78becf18bfe0599d54d02c5318d317ad8aa0fba100eab1f29b292ee585c5"} Sep 08 02:05:18.246228 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:18.246193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" event={"ID":"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb","Type":"ContainerStarted","Data":"002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e"} Sep 08 02:05:18.246354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:18.246338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:18.267657 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:18.267615 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" podStartSLOduration=1.5619387169999999 podStartE2EDuration="4.267600509s" totalImagesPullingTime="2.705661792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:18.266777324 +0000 UTC m=+399.444089399" watchObservedRunningTime="2026-09-08 02:05:18.267600509 +0000 UTC m=+399.444912580" Sep 08 02:05:20.259106 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.259005 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" event={"ID":"77ebc803-9570-4ade-9655-2a0bb00d86cc","Type":"ContainerStarted","Data":"700446c29ea9dfedc9f99cc24418f38853291c116388844cdfc45d57c8d6d7d3"} Sep 08 02:05:20.259554 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.259117 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:20.282418 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.282378 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" podStartSLOduration=2.250660238 podStartE2EDuration="4.282362951s" totalImagesPullingTime="2.031702713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:20.279384813 +0000 UTC m=+401.456696886" watchObservedRunningTime="2026-09-08 02:05:20.282362951 +0000 UTC m=+401.459675023" Sep 08 02:05:20.867197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.867161 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7"] Sep 08 02:05:20.870617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.870602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:20.872877 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.872861 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mc6hv\"" Sep 08 02:05:20.884284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.884263 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7"] Sep 08 02:05:20.959237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.959203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqqc\" (UniqueName: \"kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:20.959399 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:20.959247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.060004 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.059971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqqc\" (UniqueName: \"kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.060153 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.060019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.060475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.060455 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.068691 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.068670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqqc\" (UniqueName: \"kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc\") pod \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.180410 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.180386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:21.457341 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:21.457276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7"] Sep 08 02:05:21.459456 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:21.459424 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4579f0d3_5ddc_4251_b9b7_d5ca25603849.slice/crio-fd72cdda062a4c4bbc8ab9e143fc401d7aaa78274c184065e75da367c1927cbd WatchSource:0}: Error finding container fd72cdda062a4c4bbc8ab9e143fc401d7aaa78274c184065e75da367c1927cbd: Status 404 returned error can't find the container with id fd72cdda062a4c4bbc8ab9e143fc401d7aaa78274c184065e75da367c1927cbd Sep 08 02:05:22.272105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:22.272064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" event={"ID":"4579f0d3-5ddc-4251-b9b7-d5ca25603849","Type":"ContainerStarted","Data":"fd72cdda062a4c4bbc8ab9e143fc401d7aaa78274c184065e75da367c1927cbd"} Sep 08 02:05:23.503393 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.503355 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zv6c5"] Sep 08 02:05:23.506790 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.506774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:23.511867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.511844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6mdbt\"" Sep 08 02:05:23.520860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.520836 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zv6c5"] Sep 08 02:05:23.582000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.581960 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdfs\" (UniqueName: \"kubernetes.io/projected/daf6e428-2030-4b59-a818-3c3edfd23c75-kube-api-access-zpdfs\") pod \"authorino-operator-68d88948d8-zv6c5\" (UID: \"daf6e428-2030-4b59-a818-3c3edfd23c75\") " pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:23.682684 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.682653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdfs\" (UniqueName: \"kubernetes.io/projected/daf6e428-2030-4b59-a818-3c3edfd23c75-kube-api-access-zpdfs\") pod \"authorino-operator-68d88948d8-zv6c5\" (UID: \"daf6e428-2030-4b59-a818-3c3edfd23c75\") " pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:23.691028 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.691008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdfs\" (UniqueName: \"kubernetes.io/projected/daf6e428-2030-4b59-a818-3c3edfd23c75-kube-api-access-zpdfs\") pod \"authorino-operator-68d88948d8-zv6c5\" (UID: \"daf6e428-2030-4b59-a818-3c3edfd23c75\") " pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:23.818472 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:23.818390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:24.094530 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:24.094502 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zv6c5"] Sep 08 02:05:24.096475 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:24.096436 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf6e428_2030_4b59_a818_3c3edfd23c75.slice/crio-1dba70d0889d34fe2b26aab960067a86ea5cc5cc65e34c860204cea8d3fdac68 WatchSource:0}: Error finding container 1dba70d0889d34fe2b26aab960067a86ea5cc5cc65e34c860204cea8d3fdac68: Status 404 returned error can't find the container with id 1dba70d0889d34fe2b26aab960067a86ea5cc5cc65e34c860204cea8d3fdac68 Sep 08 02:05:24.283338 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:24.283300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" event={"ID":"daf6e428-2030-4b59-a818-3c3edfd23c75","Type":"ContainerStarted","Data":"1dba70d0889d34fe2b26aab960067a86ea5cc5cc65e34c860204cea8d3fdac68"} Sep 08 02:05:26.297889 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:26.297849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" event={"ID":"4579f0d3-5ddc-4251-b9b7-d5ca25603849","Type":"ContainerStarted","Data":"82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40"} Sep 08 02:05:26.298327 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:26.297945 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:26.317654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:26.317601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" podStartSLOduration=1.881401866 podStartE2EDuration="6.31758396s" totalImagesPullingTime="4.436182094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:26.315073052 +0000 UTC m=+407.492385126" watchObservedRunningTime="2026-09-08 02:05:26.31758396 +0000 UTC m=+407.494896032" Sep 08 02:05:27.205559 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.205527 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77547f95bc-gcv68" podUID="1e99fa06-e18b-4029-a96d-1c7efda62529" containerName="console" containerID="cri-o://b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0" gracePeriod=15 Sep 08 02:05:27.458508 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.458421 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:05:27.517105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517077 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517121 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517149 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wrbd\" (UniqueName: \"kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517204 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517324 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert\") pod \"1e99fa06-e18b-4029-a96d-1c7efda62529\" (UID: \"1e99fa06-e18b-4029-a96d-1c7efda62529\") " Sep 08 02:05:27.517873 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config" (OuterVolumeSpecName: "console-config") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:05:27.517967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517885 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:05:27.517967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517893 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:05:27.517967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.517871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:05:27.519389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.519367 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:05:27.519468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.519421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:05:27.519510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.519466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd" (OuterVolumeSpecName: "kube-api-access-9wrbd") pod "1e99fa06-e18b-4029-a96d-1c7efda62529" (UID: "1e99fa06-e18b-4029-a96d-1c7efda62529"). InnerVolumeSpecName "kube-api-access-9wrbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:27.617937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617908 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-console-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.617937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617932 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-oauth-config\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.617937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617943 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-oauth-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.618124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617951 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.618124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617960 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e99fa06-e18b-4029-a96d-1c7efda62529-trusted-ca-bundle\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.618124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617968 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrbd\" (UniqueName: \"kubernetes.io/projected/1e99fa06-e18b-4029-a96d-1c7efda62529-kube-api-access-9wrbd\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:27.618124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:27.617977 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e99fa06-e18b-4029-a96d-1c7efda62529-console-serving-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:28.311218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.311176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" event={"ID":"daf6e428-2030-4b59-a818-3c3edfd23c75","Type":"ContainerStarted","Data":"fa0a63ad46296a72f53644e8f608df2b6e0a2b8d23072bb517f56300907e52b5"} Sep 08 02:05:28.311437 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.311247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:28.312593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.312566 2559 generic.go:356] "Generic (PLEG): container finished" podID="1e99fa06-e18b-4029-a96d-1c7efda62529" containerID="b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0" exitCode=0 Sep 08 02:05:28.312709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.312604 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77547f95bc-gcv68" Sep 08 02:05:28.312709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.312625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77547f95bc-gcv68" event={"ID":"1e99fa06-e18b-4029-a96d-1c7efda62529","Type":"ContainerDied","Data":"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0"} Sep 08 02:05:28.312709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.312659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77547f95bc-gcv68" event={"ID":"1e99fa06-e18b-4029-a96d-1c7efda62529","Type":"ContainerDied","Data":"ce368bb4eeefecca1bc65a68c64b7b908e8fa855b7db46589781de0929ffff60"} Sep 08 02:05:28.312709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.312698 2559 scope.go:117] "RemoveContainer" containerID="b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0" Sep 08 02:05:28.321981 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.321961 2559 scope.go:117] "RemoveContainer" containerID="b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0" Sep 08 02:05:28.322267 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:28.322250 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0\": container with ID starting with b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0 not found: ID does not exist" containerID="b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0" Sep 08 02:05:28.322344 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.322275 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0"} err="failed to get container status \"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0\": rpc error: code = NotFound desc = could not find container \"b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0\": container with ID starting with b69c617aafd69c9b0e91b1fde1b3c08aedfd0a749dc3d08266fbe6c2d79e48d0 not found: ID does not exist" Sep 08 02:05:28.329850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.329812 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" podStartSLOduration=2.223027501 podStartE2EDuration="5.329798349s" totalImagesPullingTime="3.106770848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:28.328386274 +0000 UTC m=+409.505698347" watchObservedRunningTime="2026-09-08 02:05:28.329798349 +0000 UTC m=+409.507110421" Sep 08 02:05:28.342678 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.342650 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:05:28.346827 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.346804 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77547f95bc-gcv68"] Sep 08 02:05:28.672714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.672685 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c"] Sep 08 02:05:28.673105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.673028 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e99fa06-e18b-4029-a96d-1c7efda62529" containerName="console" Sep 08 02:05:28.673105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.673039 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e99fa06-e18b-4029-a96d-1c7efda62529" containerName="console" Sep 08 02:05:28.673105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.673095 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e99fa06-e18b-4029-a96d-1c7efda62529" containerName="console" Sep 08 02:05:28.677477 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.677458 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.679537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.679516 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 08 02:05:28.679641 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.679538 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dsb4k\"" Sep 08 02:05:28.679641 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.679544 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 08 02:05:28.685746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.685721 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c"] Sep 08 02:05:28.828735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.828699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.828735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.828735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvkc\" (UniqueName: \"kubernetes.io/projected/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-kube-api-access-htvkc\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.828936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.828808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.930035 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.929936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.930212 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.930096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.930212 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.930135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvkc\" (UniqueName: \"kubernetes.io/projected/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-kube-api-access-htvkc\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.930769 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.930752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.932425 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.932401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.937963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.937940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvkc\" (UniqueName: \"kubernetes.io/projected/0f0ddaf3-bdbe-47db-b551-7cb364d74b2f-kube-api-access-htvkc\") pod \"kuadrant-console-plugin-598f86468b-hs74c\" (UID: \"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:28.986999 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:28.986959 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" Sep 08 02:05:29.255354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:29.255250 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:29.258511 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:29.258487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c"] Sep 08 02:05:29.260985 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:29.260959 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0ddaf3_bdbe_47db_b551_7cb364d74b2f.slice/crio-f08e9d541ad77bb4f93c62a615edf959da9b3d3eee70d45ae17876fc9a5f4bcf WatchSource:0}: Error finding container f08e9d541ad77bb4f93c62a615edf959da9b3d3eee70d45ae17876fc9a5f4bcf: Status 404 returned error can't find the container with id f08e9d541ad77bb4f93c62a615edf959da9b3d3eee70d45ae17876fc9a5f4bcf Sep 08 02:05:29.321769 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:29.321737 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e99fa06-e18b-4029-a96d-1c7efda62529" path="/var/lib/kubelet/pods/1e99fa06-e18b-4029-a96d-1c7efda62529/volumes" Sep 08 02:05:29.322688 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:29.322662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" event={"ID":"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f","Type":"ContainerStarted","Data":"f08e9d541ad77bb4f93c62a615edf959da9b3d3eee70d45ae17876fc9a5f4bcf"} Sep 08 02:05:31.267225 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:31.267184 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-gg6nq" Sep 08 02:05:34.353692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:34.353653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" event={"ID":"0f0ddaf3-bdbe-47db-b551-7cb364d74b2f","Type":"ContainerStarted","Data":"3b7d4ce205d3f06e316cc30cca68db746fa496ced3503a018f74f30ef8e0eda3"} Sep 08 02:05:34.369796 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:34.369751 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hs74c" podStartSLOduration=1.775703582 podStartE2EDuration="6.369736059s" totalImagesPullingTime="4.594032477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:34.367026387 +0000 UTC m=+415.544338459" watchObservedRunningTime="2026-09-08 02:05:34.369736059 +0000 UTC m=+415.547048131" Sep 08 02:05:37.306394 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:37.306361 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:38.367807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.367772 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s"] Sep 08 02:05:38.373101 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.373078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.382942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.382907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s"] Sep 08 02:05:38.515811 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.515771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgtv\" (UniqueName: \"kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.515980 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.515828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.617011 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.616971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgtv\" (UniqueName: \"kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.617212 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.617031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.617451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.617420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.624729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.624666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgtv\" (UniqueName: \"kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv\") pod \"kuadrant-operator-controller-manager-96db98c47-clh4s\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.684528 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.684489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:38.982783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:38.982758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s"] Sep 08 02:05:38.984808 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:38.984781 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb099bff8_0699_4555_8b1f_06ae86946091.slice/crio-8b965ed8fc8c688bf60dd95d8a9fb0de9be309fae73bb555e7a921d69f24b81d WatchSource:0}: Error finding container 8b965ed8fc8c688bf60dd95d8a9fb0de9be309fae73bb555e7a921d69f24b81d: Status 404 returned error can't find the container with id 8b965ed8fc8c688bf60dd95d8a9fb0de9be309fae73bb555e7a921d69f24b81d Sep 08 02:05:39.325757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.325728 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-zv6c5" Sep 08 02:05:39.384630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.384594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" event={"ID":"b099bff8-0699-4555-8b1f-06ae86946091","Type":"ContainerStarted","Data":"941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52"} Sep 08 02:05:39.385041 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.384654 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:39.385041 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.384675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" event={"ID":"b099bff8-0699-4555-8b1f-06ae86946091","Type":"ContainerStarted","Data":"8b965ed8fc8c688bf60dd95d8a9fb0de9be309fae73bb555e7a921d69f24b81d"} Sep 08 02:05:39.403387 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.403331 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" podStartSLOduration=1.40327177 podStartE2EDuration="1.40327177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:39.402623282 +0000 UTC m=+420.579935367" watchObservedRunningTime="2026-09-08 02:05:39.40327177 +0000 UTC m=+420.580583842" Sep 08 02:05:39.417991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.417959 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7"] Sep 08 02:05:39.418148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.418124 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" containerName="manager" containerID="cri-o://82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40" gracePeriod=2 Sep 08 02:05:39.436352 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.434406 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7"] Sep 08 02:05:39.439211 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.439167 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4579f0d3-5ddc-4251-b9b7-d5ca25603849\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-6vbq7\": pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.446663 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.446640 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s"] Sep 08 02:05:39.457660 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.457631 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s"] Sep 08 02:05:39.471015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.470961 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7"] Sep 08 02:05:39.471222 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.471191 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" containerName="manager" containerID="cri-o://002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e" gracePeriod=2 Sep 08 02:05:39.473312 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.473262 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.474365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474344 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:05:39.474757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474742 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" containerName="manager" Sep 08 02:05:39.474848 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474759 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" containerName="manager" Sep 08 02:05:39.474848 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474774 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b099bff8-0699-4555-8b1f-06ae86946091" containerName="manager" Sep 08 02:05:39.474848 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474782 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b099bff8-0699-4555-8b1f-06ae86946091" containerName="manager" Sep 08 02:05:39.475662 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474953 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b099bff8-0699-4555-8b1f-06ae86946091" containerName="manager" Sep 08 02:05:39.475662 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.474970 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" containerName="manager" Sep 08 02:05:39.479239 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.479176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.501968 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.501491 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-wnxn7\": pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.503683 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.503634 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.504881 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.504017 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7"] Sep 08 02:05:39.507420 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.507207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:05:39.510510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.510459 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd"] Sep 08 02:05:39.511040 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.510928 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" containerName="manager" Sep 08 02:05:39.511040 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.510947 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" containerName="manager" Sep 08 02:05:39.511234 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.511059 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" containerName="manager" Sep 08 02:05:39.517380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.517361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:39.529208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.529182 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd"] Sep 08 02:05:39.537570 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.537533 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.539368 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.539342 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.556941 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.556865 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.624304 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.624256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvgz\" (UniqueName: \"kubernetes.io/projected/1425544c-22ef-4323-9a0c-cf99361813fe-kube-api-access-2jvgz\") pod \"limitador-operator-controller-manager-6c549b6d85-wrfcd\" (UID: \"1425544c-22ef-4323-9a0c-cf99361813fe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:39.624451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.624332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.624451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.624364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddwt\" (UniqueName: \"kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.721196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.721176 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.723344 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.723318 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.724006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.723987 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.725134 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.725108 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.725280 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.725262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddwt\" (UniqueName: \"kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.725364 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.725339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvgz\" (UniqueName: \"kubernetes.io/projected/1425544c-22ef-4323-9a0c-cf99361813fe-kube-api-access-2jvgz\") pod \"limitador-operator-controller-manager-6c549b6d85-wrfcd\" (UID: \"1425544c-22ef-4323-9a0c-cf99361813fe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:39.725422 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.725393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.725734 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.725716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.728228 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.728203 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:39.729766 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:39.729744 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:39.734190 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.734168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvgz\" (UniqueName: \"kubernetes.io/projected/1425544c-22ef-4323-9a0c-cf99361813fe-kube-api-access-2jvgz\") pod \"limitador-operator-controller-manager-6c549b6d85-wrfcd\" (UID: \"1425544c-22ef-4323-9a0c-cf99361813fe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:39.734516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.734500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddwt\" (UniqueName: \"kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt\") pod \"kuadrant-operator-controller-manager-96db98c47-2k4f9\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.826689 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.826610 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8clm\" (UniqueName: \"kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm\") pod \"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\" (UID: \"9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb\") " Sep 08 02:05:39.826821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.826703 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wqqc\" (UniqueName: \"kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc\") pod \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " Sep 08 02:05:39.826821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.826796 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume\") pod \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\" (UID: \"4579f0d3-5ddc-4251-b9b7-d5ca25603849\") " Sep 08 02:05:39.827165 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.827137 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4579f0d3-5ddc-4251-b9b7-d5ca25603849" (UID: "4579f0d3-5ddc-4251-b9b7-d5ca25603849"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:39.828834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.828809 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm" (OuterVolumeSpecName: "kube-api-access-n8clm") pod "9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" (UID: "9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb"). InnerVolumeSpecName "kube-api-access-n8clm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:39.828834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.828817 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc" (OuterVolumeSpecName: "kube-api-access-2wqqc") pod "4579f0d3-5ddc-4251-b9b7-d5ca25603849" (UID: "4579f0d3-5ddc-4251-b9b7-d5ca25603849"). InnerVolumeSpecName "kube-api-access-2wqqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:39.869161 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.869132 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:39.874080 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.874054 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:39.927597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.927567 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqqc\" (UniqueName: \"kubernetes.io/projected/4579f0d3-5ddc-4251-b9b7-d5ca25603849-kube-api-access-2wqqc\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:39.927717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.927602 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4579f0d3-5ddc-4251-b9b7-d5ca25603849-extensions-socket-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:39.927717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:39.927618 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8clm\" (UniqueName: \"kubernetes.io/projected/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb-kube-api-access-n8clm\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:40.169686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.169658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd"] Sep 08 02:05:40.171344 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:40.171275 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1425544c_22ef_4323_9a0c_cf99361813fe.slice/crio-b3e5a8e206f2f2ab071f997f58714928be2c097644043a802365085329740dde WatchSource:0}: Error finding container b3e5a8e206f2f2ab071f997f58714928be2c097644043a802365085329740dde: Status 404 returned error can't find the container with id b3e5a8e206f2f2ab071f997f58714928be2c097644043a802365085329740dde Sep 08 02:05:40.174786 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.174591 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:05:40.177476 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:40.177452 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e48b54_02a3_4efe_937d_102bba34d293.slice/crio-1f77ab45985084ec6c646a090b74af5a5d3f6b2bc490aa24e05abd344068a0e3 WatchSource:0}: Error finding container 1f77ab45985084ec6c646a090b74af5a5d3f6b2bc490aa24e05abd344068a0e3: Status 404 returned error can't find the container with id 1f77ab45985084ec6c646a090b74af5a5d3f6b2bc490aa24e05abd344068a0e3 Sep 08 02:05:40.392189 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.392149 2559 generic.go:356] "Generic (PLEG): container finished" podID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" containerID="82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40" exitCode=0 Sep 08 02:05:40.392189 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.392186 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:40.392733 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.392244 2559 scope.go:117] "RemoveContainer" containerID="82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40" Sep 08 02:05:40.394575 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.394546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" event={"ID":"e5e48b54-02a3-4efe-937d-102bba34d293","Type":"ContainerStarted","Data":"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0"} Sep 08 02:05:40.394575 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.394553 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:40.394771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.394592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" event={"ID":"e5e48b54-02a3-4efe-937d-102bba34d293","Type":"ContainerStarted","Data":"1f77ab45985084ec6c646a090b74af5a5d3f6b2bc490aa24e05abd344068a0e3"} Sep 08 02:05:40.394771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.394661 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:40.396334 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.396278 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:40.396621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.396532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" event={"ID":"1425544c-22ef-4323-9a0c-cf99361813fe","Type":"ContainerStarted","Data":"8a74aabc0663741ffa1268be8aa709a9ceb409ec600d5ecb6a18cc7f426f32f0"} Sep 08 02:05:40.396621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.396593 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" event={"ID":"1425544c-22ef-4323-9a0c-cf99361813fe","Type":"ContainerStarted","Data":"b3e5a8e206f2f2ab071f997f58714928be2c097644043a802365085329740dde"} Sep 08 02:05:40.396621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.396622 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:40.398002 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.397972 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:40.398489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.398467 2559 generic.go:356] "Generic (PLEG): container finished" podID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" containerID="002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e" exitCode=0 Sep 08 02:05:40.398577 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.398510 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:40.398864 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.398845 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" container="manager" Sep 08 02:05:40.403859 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.403842 2559 scope.go:117] "RemoveContainer" containerID="82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40" Sep 08 02:05:40.404150 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.404132 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40\": container with ID starting with 82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40 not found: ID does not exist" containerID="82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40" Sep 08 02:05:40.404227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.404158 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40"} err="failed to get container status \"82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40\": rpc error: code = NotFound desc = could not find container \"82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40\": container with ID starting with 82043f3af763abd55d62979e4af5eeefb2ac1ddd2e68f9295de903726a568c40 not found: ID does not exist" Sep 08 02:05:40.404227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.404176 2559 scope.go:117] "RemoveContainer" containerID="002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e" Sep 08 02:05:40.413388 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.413369 2559 scope.go:117] "RemoveContainer" containerID="002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e" Sep 08 02:05:40.413646 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.413625 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e\": container with ID starting with 002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e not found: ID does not exist" containerID="002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e" Sep 08 02:05:40.413730 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.413648 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e"} err="failed to get container status \"002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e\": rpc error: code = NotFound desc = could not find container \"002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e\": container with ID starting with 002502eee353d464e6d4f003cb21c9c712886aeef51d57893c23abb9f8cf172e not found: ID does not exist" Sep 08 02:05:40.421868 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.421834 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" podStartSLOduration=1.421821586 podStartE2EDuration="1.421821586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:40.419535456 +0000 UTC m=+421.596847527" watchObservedRunningTime="2026-09-08 02:05:40.421821586 +0000 UTC m=+421.599133660" Sep 08 02:05:40.422020 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.421998 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:40.444038 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:40.443995 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" podStartSLOduration=1.443982152 podStartE2EDuration="1.443982152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:40.442837248 +0000 UTC m=+421.620149320" watchObservedRunningTime="2026-09-08 02:05:40.443982152 +0000 UTC m=+421.621294225" Sep 08 02:05:40.444552 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.444527 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-6vbq7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-6vbq7" Sep 08 02:05:40.445974 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.445955 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wnxn7\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wnxn7" Sep 08 02:05:40.447528 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:40.447504 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-clh4s\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="b099bff8-0699-4555-8b1f-06ae86946091" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:41.320014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.319978 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4579f0d3-5ddc-4251-b9b7-d5ca25603849" path="/var/lib/kubelet/pods/4579f0d3-5ddc-4251-b9b7-d5ca25603849/volumes" Sep 08 02:05:41.320457 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.320430 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb" path="/var/lib/kubelet/pods/9da2d30f-47e7-4ef0-ac21-f1b82ab4c0bb/volumes" Sep 08 02:05:41.407204 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.407162 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" podUID="b099bff8-0699-4555-8b1f-06ae86946091" containerName="manager" containerID="cri-o://941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52" gracePeriod=2 Sep 08 02:05:41.659164 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.659131 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:41.843369 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.843248 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume\") pod \"b099bff8-0699-4555-8b1f-06ae86946091\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " Sep 08 02:05:41.843546 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.843392 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgtv\" (UniqueName: \"kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv\") pod \"b099bff8-0699-4555-8b1f-06ae86946091\" (UID: \"b099bff8-0699-4555-8b1f-06ae86946091\") " Sep 08 02:05:41.843674 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.843650 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b099bff8-0699-4555-8b1f-06ae86946091" (UID: "b099bff8-0699-4555-8b1f-06ae86946091"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:05:41.845455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.845428 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv" (OuterVolumeSpecName: "kube-api-access-hcgtv") pod "b099bff8-0699-4555-8b1f-06ae86946091" (UID: "b099bff8-0699-4555-8b1f-06ae86946091"). InnerVolumeSpecName "kube-api-access-hcgtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:05:41.944051 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.944019 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgtv\" (UniqueName: \"kubernetes.io/projected/b099bff8-0699-4555-8b1f-06ae86946091-kube-api-access-hcgtv\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:41.944051 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:41.944048 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b099bff8-0699-4555-8b1f-06ae86946091-extensions-socket-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:05:42.414713 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:42.414676 2559 generic.go:356] "Generic (PLEG): container finished" podID="b099bff8-0699-4555-8b1f-06ae86946091" containerID="941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52" exitCode=0 Sep 08 02:05:42.415155 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:42.414717 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:42.415155 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:42.414776 2559 scope.go:117] "RemoveContainer" containerID="941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52" Sep 08 02:05:42.416916 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:42.416893 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-clh4s\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="b099bff8-0699-4555-8b1f-06ae86946091" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:42.429268 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:42.429234 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-clh4s\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="b099bff8-0699-4555-8b1f-06ae86946091" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-clh4s" Sep 08 02:05:42.433261 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:42.433241 2559 scope.go:117] "RemoveContainer" containerID="941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52" Sep 08 02:05:42.433623 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:05:42.433598 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52\": container with ID starting with 941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52 not found: ID does not exist" containerID="941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52" Sep 08 02:05:42.433686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:42.433630 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52"} err="failed to get container status \"941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52\": rpc error: code = NotFound desc = could not find container \"941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52\": container with ID starting with 941a9ec504e2061da87f374e41985974981af1cd856ad940a0536115cd012c52 not found: ID does not exist" Sep 08 02:05:43.319469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:43.319438 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b099bff8-0699-4555-8b1f-06ae86946091" path="/var/lib/kubelet/pods/b099bff8-0699-4555-8b1f-06ae86946091/volumes" Sep 08 02:05:51.410913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.410883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:05:51.411426 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.410956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wrfcd" Sep 08 02:05:51.762148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.762074 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:05:51.767585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.767567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.792109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.792075 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:05:51.821706 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.821678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qss85\" (UniqueName: \"kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.821844 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.821761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.923107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.923069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.923277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.923122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qss85\" (UniqueName: \"kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.923469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.923449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:51.934179 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:51.934145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qss85\" (UniqueName: \"kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-n9crn\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:52.078924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.078844 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:52.359825 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.359746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:05:52.363115 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:05:52.363087 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd85030_540f_471c_a672_24d0d1b8fdd0.slice/crio-2e51bbc31d7db8e4e7dc90934083a3e3f97d7a859ff60877f4f1121857664902 WatchSource:0}: Error finding container 2e51bbc31d7db8e4e7dc90934083a3e3f97d7a859ff60877f4f1121857664902: Status 404 returned error can't find the container with id 2e51bbc31d7db8e4e7dc90934083a3e3f97d7a859ff60877f4f1121857664902 Sep 08 02:05:52.471927 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.471888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" event={"ID":"cdd85030-540f-471c-a672-24d0d1b8fdd0","Type":"ContainerStarted","Data":"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa"} Sep 08 02:05:52.472348 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.471954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" event={"ID":"cdd85030-540f-471c-a672-24d0d1b8fdd0","Type":"ContainerStarted","Data":"2e51bbc31d7db8e4e7dc90934083a3e3f97d7a859ff60877f4f1121857664902"} Sep 08 02:05:52.472348 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.471998 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:05:52.493823 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:05:52.493773 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" podStartSLOduration=1.493757223 podStartE2EDuration="1.493757223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:05:52.493170873 +0000 UTC m=+433.670482953" watchObservedRunningTime="2026-09-08 02:05:52.493757223 +0000 UTC m=+433.671069295" Sep 08 02:06:03.480757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.480685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:06:03.532950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.532914 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:06:03.533216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.533166 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" podUID="e5e48b54-02a3-4efe-937d-102bba34d293" containerName="manager" containerID="cri-o://f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0" gracePeriod=10 Sep 08 02:06:03.785971 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.785943 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:06:03.811147 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.811117 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hddwt\" (UniqueName: \"kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt\") pod \"e5e48b54-02a3-4efe-937d-102bba34d293\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " Sep 08 02:06:03.811147 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.811151 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume\") pod \"e5e48b54-02a3-4efe-937d-102bba34d293\" (UID: \"e5e48b54-02a3-4efe-937d-102bba34d293\") " Sep 08 02:06:03.811711 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.811671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e5e48b54-02a3-4efe-937d-102bba34d293" (UID: "e5e48b54-02a3-4efe-937d-102bba34d293"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:06:03.813170 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.813147 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt" (OuterVolumeSpecName: "kube-api-access-hddwt") pod "e5e48b54-02a3-4efe-937d-102bba34d293" (UID: "e5e48b54-02a3-4efe-937d-102bba34d293"). InnerVolumeSpecName "kube-api-access-hddwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:06:03.912298 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.912247 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hddwt\" (UniqueName: \"kubernetes.io/projected/e5e48b54-02a3-4efe-937d-102bba34d293-kube-api-access-hddwt\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:03.912298 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:03.912281 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5e48b54-02a3-4efe-937d-102bba34d293-extensions-socket-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:04.540409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.540370 2559 generic.go:356] "Generic (PLEG): container finished" podID="e5e48b54-02a3-4efe-937d-102bba34d293" containerID="f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0" exitCode=0 Sep 08 02:06:04.540788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.540416 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" Sep 08 02:06:04.540788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.540457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" event={"ID":"e5e48b54-02a3-4efe-937d-102bba34d293","Type":"ContainerDied","Data":"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0"} Sep 08 02:06:04.540788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.540519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9" event={"ID":"e5e48b54-02a3-4efe-937d-102bba34d293","Type":"ContainerDied","Data":"1f77ab45985084ec6c646a090b74af5a5d3f6b2bc490aa24e05abd344068a0e3"} Sep 08 02:06:04.540788 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.540542 2559 scope.go:117] "RemoveContainer" containerID="f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0" Sep 08 02:06:04.549627 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.549604 2559 scope.go:117] "RemoveContainer" containerID="f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0" Sep 08 02:06:04.549918 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:06:04.549898 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0\": container with ID starting with f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0 not found: ID does not exist" containerID="f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0" Sep 08 02:06:04.549977 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.549926 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0"} err="failed to get container status \"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0\": rpc error: code = NotFound desc = could not find container \"f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0\": container with ID starting with f89732fad36e3ed332477e5e72dc9b4dbb837d261b200ae98234c8964e8bd7d0 not found: ID does not exist" Sep 08 02:06:04.563490 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.563463 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:06:04.567115 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:04.567091 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2k4f9"] Sep 08 02:06:05.319109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:05.319075 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e48b54-02a3-4efe-937d-102bba34d293" path="/var/lib/kubelet/pods/e5e48b54-02a3-4efe-937d-102bba34d293/volumes" Sep 08 02:06:23.463019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.462984 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:23.463456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.463384 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e48b54-02a3-4efe-937d-102bba34d293" containerName="manager" Sep 08 02:06:23.463456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.463396 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e48b54-02a3-4efe-937d-102bba34d293" containerName="manager" Sep 08 02:06:23.463456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.463458 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e48b54-02a3-4efe-937d-102bba34d293" containerName="manager" Sep 08 02:06:23.485478 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.485445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:23.485640 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.485586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.488085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.488056 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 02:06:23.568155 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.568122 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:23.576926 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.576895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrvr\" (UniqueName: \"kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.577063 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.576930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.677805 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.677765 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrvr\" (UniqueName: \"kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.677997 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.677823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.678701 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.678680 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.685564 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.685542 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrvr\" (UniqueName: \"kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr\") pod \"limitador-limitador-859c4d96d6-zs8dr\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:23.795392 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:23.795281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:24.069268 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.069238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:24.071007 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:06:24.070977 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33ab29_243b_403f_95e3_5897b055fd4f.slice/crio-8eb9b743e1570f37f1c9913b2c2f56b5724c1402865f3a87ff0a14cce539c407 WatchSource:0}: Error finding container 8eb9b743e1570f37f1c9913b2c2f56b5724c1402865f3a87ff0a14cce539c407: Status 404 returned error can't find the container with id 8eb9b743e1570f37f1c9913b2c2f56b5724c1402865f3a87ff0a14cce539c407 Sep 08 02:06:24.258516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.258485 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:24.263329 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.263283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:24.265793 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.265764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-md94z\"" Sep 08 02:06:24.268378 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.268351 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:24.383778 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.383687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx85j\" (UniqueName: \"kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j\") pod \"authorino-b54df94b4-klwgv\" (UID: \"d7690d60-41f2-4c63-b884-4265ee704089\") " pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:24.466637 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.466599 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:06:24.470153 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.470129 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:06:24.474481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.474458 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:06:24.484812 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.484785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx85j\" (UniqueName: \"kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j\") pod \"authorino-b54df94b4-klwgv\" (UID: \"d7690d60-41f2-4c63-b884-4265ee704089\") " pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:24.492651 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.492614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx85j\" (UniqueName: \"kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j\") pod \"authorino-b54df94b4-klwgv\" (UID: \"d7690d60-41f2-4c63-b884-4265ee704089\") " pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:24.575320 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.575257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:24.585613 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.585581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbmt\" (UniqueName: \"kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt\") pod \"authorino-c548b5d7d-nj7mn\" (UID: \"e4a5ae43-b8e7-4627-a868-7f42b2a79404\") " pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:06:24.654099 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.654068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" event={"ID":"6d33ab29-243b-403f-95e3-5897b055fd4f","Type":"ContainerStarted","Data":"8eb9b743e1570f37f1c9913b2c2f56b5724c1402865f3a87ff0a14cce539c407"} Sep 08 02:06:24.686892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.686856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbmt\" (UniqueName: \"kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt\") pod \"authorino-c548b5d7d-nj7mn\" (UID: \"e4a5ae43-b8e7-4627-a868-7f42b2a79404\") " pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:06:24.698415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.698385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbmt\" (UniqueName: \"kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt\") pod \"authorino-c548b5d7d-nj7mn\" (UID: \"e4a5ae43-b8e7-4627-a868-7f42b2a79404\") " pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:06:24.781425 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.781385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:06:24.844959 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:24.844878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:24.847445 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:06:24.847418 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7690d60_41f2_4c63_b884_4265ee704089.slice/crio-9c25f7899d16380713d4d214efb5c720b3e08187cff0e8454cc3e86087c8ab44 WatchSource:0}: Error finding container 9c25f7899d16380713d4d214efb5c720b3e08187cff0e8454cc3e86087c8ab44: Status 404 returned error can't find the container with id 9c25f7899d16380713d4d214efb5c720b3e08187cff0e8454cc3e86087c8ab44 Sep 08 02:06:25.052374 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.052349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:06:25.054243 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:06:25.054219 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4a5ae43_b8e7_4627_a868_7f42b2a79404.slice/crio-0488d34a7f85f713e4e8b1ae5ae7d6c27ad590a2111183e5e86ea7d888d1818d WatchSource:0}: Error finding container 0488d34a7f85f713e4e8b1ae5ae7d6c27ad590a2111183e5e86ea7d888d1818d: Status 404 returned error can't find the container with id 0488d34a7f85f713e4e8b1ae5ae7d6c27ad590a2111183e5e86ea7d888d1818d Sep 08 02:06:25.663153 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.663078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-klwgv" event={"ID":"d7690d60-41f2-4c63-b884-4265ee704089","Type":"ContainerStarted","Data":"9c25f7899d16380713d4d214efb5c720b3e08187cff0e8454cc3e86087c8ab44"} Sep 08 02:06:25.664840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.664804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" event={"ID":"e4a5ae43-b8e7-4627-a868-7f42b2a79404","Type":"ContainerStarted","Data":"0488d34a7f85f713e4e8b1ae5ae7d6c27ad590a2111183e5e86ea7d888d1818d"} Sep 08 02:06:25.666751 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.666706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" event={"ID":"6d33ab29-243b-403f-95e3-5897b055fd4f","Type":"ContainerStarted","Data":"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7"} Sep 08 02:06:25.666861 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.666834 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:25.682180 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:25.682081 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" podStartSLOduration=1.174666691 podStartE2EDuration="2.682059069s" totalImagesPullingTime="1.507392378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:06:25.68069689 +0000 UTC m=+466.858008959" watchObservedRunningTime="2026-09-08 02:06:25.682059069 +0000 UTC m=+466.859371143" Sep 08 02:06:27.682466 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:27.682430 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" event={"ID":"e4a5ae43-b8e7-4627-a868-7f42b2a79404","Type":"ContainerStarted","Data":"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6"} Sep 08 02:06:27.684026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:27.683986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-klwgv" event={"ID":"d7690d60-41f2-4c63-b884-4265ee704089","Type":"ContainerStarted","Data":"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f"} Sep 08 02:06:27.696389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:27.696350 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" podStartSLOduration=1.62247895 podStartE2EDuration="3.69633774s" totalImagesPullingTime="2.07385879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:06:27.694927915 +0000 UTC m=+468.872239986" watchObservedRunningTime="2026-09-08 02:06:27.69633774 +0000 UTC m=+468.873649803" Sep 08 02:06:27.708553 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:27.708514 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-klwgv" podStartSLOduration=1.416219385 podStartE2EDuration="3.708497983s" totalImagesPullingTime="2.292278598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:06:27.707333002 +0000 UTC m=+468.884645076" watchObservedRunningTime="2026-09-08 02:06:27.708497983 +0000 UTC m=+468.885810032" Sep 08 02:06:27.728877 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:27.728849 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:29.695427 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:29.695390 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-klwgv" podUID="d7690d60-41f2-4c63-b884-4265ee704089" containerName="authorino" containerID="cri-o://546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f" gracePeriod=30 Sep 08 02:06:29.943665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:29.943644 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:30.036412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.036336 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx85j\" (UniqueName: \"kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j\") pod \"d7690d60-41f2-4c63-b884-4265ee704089\" (UID: \"d7690d60-41f2-4c63-b884-4265ee704089\") " Sep 08 02:06:30.038256 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.038231 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j" (OuterVolumeSpecName: "kube-api-access-zx85j") pod "d7690d60-41f2-4c63-b884-4265ee704089" (UID: "d7690d60-41f2-4c63-b884-4265ee704089"). InnerVolumeSpecName "kube-api-access-zx85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:06:30.137787 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.137762 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx85j\" (UniqueName: \"kubernetes.io/projected/d7690d60-41f2-4c63-b884-4265ee704089-kube-api-access-zx85j\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:30.702155 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.702123 2559 generic.go:356] "Generic (PLEG): container finished" podID="d7690d60-41f2-4c63-b884-4265ee704089" containerID="546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f" exitCode=0 Sep 08 02:06:30.702635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.702165 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-klwgv" Sep 08 02:06:30.702635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.702207 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-klwgv" event={"ID":"d7690d60-41f2-4c63-b884-4265ee704089","Type":"ContainerDied","Data":"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f"} Sep 08 02:06:30.702635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.702258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-klwgv" event={"ID":"d7690d60-41f2-4c63-b884-4265ee704089","Type":"ContainerDied","Data":"9c25f7899d16380713d4d214efb5c720b3e08187cff0e8454cc3e86087c8ab44"} Sep 08 02:06:30.702635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.702281 2559 scope.go:117] "RemoveContainer" containerID="546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f" Sep 08 02:06:30.711240 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.711223 2559 scope.go:117] "RemoveContainer" containerID="546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f" Sep 08 02:06:30.711512 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:06:30.711495 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f\": container with ID starting with 546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f not found: ID does not exist" containerID="546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f" Sep 08 02:06:30.711565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.711521 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f"} err="failed to get container status \"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f\": rpc error: code = NotFound desc = could not find container \"546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f\": container with ID starting with 546016bed6afdc3bc8faf2568649c402cd4a5b7547fad5156633079caf7a360f not found: ID does not exist" Sep 08 02:06:30.721847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.721825 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:30.728138 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:30.728118 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-klwgv"] Sep 08 02:06:31.326137 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:31.326103 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7690d60-41f2-4c63-b884-4265ee704089" path="/var/lib/kubelet/pods/d7690d60-41f2-4c63-b884-4265ee704089/volumes" Sep 08 02:06:36.676214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:36.676176 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:37.392462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:37.392429 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:37.392670 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:37.392646 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" podUID="6d33ab29-243b-403f-95e3-5897b055fd4f" containerName="limitador" containerID="cri-o://0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7" gracePeriod=30 Sep 08 02:06:37.941746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:37.941720 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:38.006148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.006076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file\") pod \"6d33ab29-243b-403f-95e3-5897b055fd4f\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " Sep 08 02:06:38.006148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.006133 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zrvr\" (UniqueName: \"kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr\") pod \"6d33ab29-243b-403f-95e3-5897b055fd4f\" (UID: \"6d33ab29-243b-403f-95e3-5897b055fd4f\") " Sep 08 02:06:38.006412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.006388 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file" (OuterVolumeSpecName: "config-file") pod "6d33ab29-243b-403f-95e3-5897b055fd4f" (UID: "6d33ab29-243b-403f-95e3-5897b055fd4f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:06:38.008153 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.008132 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr" (OuterVolumeSpecName: "kube-api-access-4zrvr") pod "6d33ab29-243b-403f-95e3-5897b055fd4f" (UID: "6d33ab29-243b-403f-95e3-5897b055fd4f"). InnerVolumeSpecName "kube-api-access-4zrvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:06:38.107278 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.107247 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d33ab29-243b-403f-95e3-5897b055fd4f-config-file\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:38.107278 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.107271 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrvr\" (UniqueName: \"kubernetes.io/projected/6d33ab29-243b-403f-95e3-5897b055fd4f-kube-api-access-4zrvr\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:38.748815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.748781 2559 generic.go:356] "Generic (PLEG): container finished" podID="6d33ab29-243b-403f-95e3-5897b055fd4f" containerID="0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7" exitCode=0 Sep 08 02:06:38.748991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.748838 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" Sep 08 02:06:38.748991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.748866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" event={"ID":"6d33ab29-243b-403f-95e3-5897b055fd4f","Type":"ContainerDied","Data":"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7"} Sep 08 02:06:38.748991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.748956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zs8dr" event={"ID":"6d33ab29-243b-403f-95e3-5897b055fd4f","Type":"ContainerDied","Data":"8eb9b743e1570f37f1c9913b2c2f56b5724c1402865f3a87ff0a14cce539c407"} Sep 08 02:06:38.748991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.748984 2559 scope.go:117] "RemoveContainer" containerID="0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7" Sep 08 02:06:38.759160 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.759142 2559 scope.go:117] "RemoveContainer" containerID="0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7" Sep 08 02:06:38.759418 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:06:38.759399 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7\": container with ID starting with 0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7 not found: ID does not exist" containerID="0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7" Sep 08 02:06:38.759481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.759426 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7"} err="failed to get container status \"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7\": rpc error: code = NotFound desc = could not find container \"0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7\": container with ID starting with 0fb5b0d4c74eb21f6fcb6819f0ab606d4fa96cc3f1110a237e7806d7633364f7 not found: ID does not exist" Sep 08 02:06:38.771828 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.771803 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:38.775519 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:38.775500 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zs8dr"] Sep 08 02:06:39.320043 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:39.320018 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d33ab29-243b-403f-95e3-5897b055fd4f" path="/var/lib/kubelet/pods/6d33ab29-243b-403f-95e3-5897b055fd4f/volumes" Sep 08 02:06:42.082306 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc985"] Sep 08 02:06:42.082672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082620 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7690d60-41f2-4c63-b884-4265ee704089" containerName="authorino" Sep 08 02:06:42.082672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082632 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7690d60-41f2-4c63-b884-4265ee704089" containerName="authorino" Sep 08 02:06:42.082672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082655 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d33ab29-243b-403f-95e3-5897b055fd4f" containerName="limitador" Sep 08 02:06:42.082672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082660 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d33ab29-243b-403f-95e3-5897b055fd4f" containerName="limitador" Sep 08 02:06:42.082806 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082713 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7690d60-41f2-4c63-b884-4265ee704089" containerName="authorino" Sep 08 02:06:42.082806 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.082725 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d33ab29-243b-403f-95e3-5897b055fd4f" containerName="limitador" Sep 08 02:06:42.087432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.087408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.090088 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.090066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 02:06:42.090088 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.090079 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 08 02:06:42.090310 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.090140 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 02:06:42.090732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.090715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vjz6r\"" Sep 08 02:06:42.100686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.100661 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc985"] Sep 08 02:06:42.241524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.241481 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4jw\" (UniqueName: \"kubernetes.io/projected/b921561f-5241-4845-b6f7-7c7d66593fb8-kube-api-access-ds4jw\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.241717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.241551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b921561f-5241-4845-b6f7-7c7d66593fb8-data\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.342776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.342687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4jw\" (UniqueName: \"kubernetes.io/projected/b921561f-5241-4845-b6f7-7c7d66593fb8-kube-api-access-ds4jw\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.342776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.342764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b921561f-5241-4845-b6f7-7c7d66593fb8-data\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.343076 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.343061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b921561f-5241-4845-b6f7-7c7d66593fb8-data\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.350586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.350553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4jw\" (UniqueName: \"kubernetes.io/projected/b921561f-5241-4845-b6f7-7c7d66593fb8-kube-api-access-ds4jw\") pod \"postgres-69fb4594cf-mc985\" (UID: \"b921561f-5241-4845-b6f7-7c7d66593fb8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.404462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.404423 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:42.674905 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.674878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc985"] Sep 08 02:06:42.676799 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:06:42.676761 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb921561f_5241_4845_b6f7_7c7d66593fb8.slice/crio-23700987248cbb20673c7857e1d4dd16e926f6a55f7e866ca21b97f0cd4dfbce WatchSource:0}: Error finding container 23700987248cbb20673c7857e1d4dd16e926f6a55f7e866ca21b97f0cd4dfbce: Status 404 returned error can't find the container with id 23700987248cbb20673c7857e1d4dd16e926f6a55f7e866ca21b97f0cd4dfbce Sep 08 02:06:42.771364 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:42.771324 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" event={"ID":"b921561f-5241-4845-b6f7-7c7d66593fb8","Type":"ContainerStarted","Data":"23700987248cbb20673c7857e1d4dd16e926f6a55f7e866ca21b97f0cd4dfbce"} Sep 08 02:06:47.804584 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:47.804550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" event={"ID":"b921561f-5241-4845-b6f7-7c7d66593fb8","Type":"ContainerStarted","Data":"f16fc7695bc5e24400fea7ceb2be4e42b812e8ef8d0461fb86ef364510dd5c2b"} Sep 08 02:06:47.805039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:47.804678 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:47.821474 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:47.821423 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" podStartSLOduration=0.912958531 podStartE2EDuration="5.821411075s" totalImagesPullingTime="4.908452544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:06:47.820718055 +0000 UTC m=+488.998030137" watchObservedRunningTime="2026-09-08 02:06:47.821411075 +0000 UTC m=+488.998723189" Sep 08 02:06:53.839255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:53.839227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc985" Sep 08 02:06:54.337633 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.337604 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kcfbl"] Sep 08 02:06:54.344465 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.344443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.351061 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.351032 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kcfbl"] Sep 08 02:06:54.454996 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.454964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwgf\" (UniqueName: \"kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf\") pod \"authorino-5dc7ccb5f-kcfbl\" (UID: \"8c2685b1-f576-44f8-8ba7-6f7e1997d424\") " pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.555402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.555366 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kcfbl"] Sep 08 02:06:54.555640 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:06:54.555616 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-srwgf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" podUID="8c2685b1-f576-44f8-8ba7-6f7e1997d424" Sep 08 02:06:54.555681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.555662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwgf\" (UniqueName: \"kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf\") pod \"authorino-5dc7ccb5f-kcfbl\" (UID: \"8c2685b1-f576-44f8-8ba7-6f7e1997d424\") " pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.565554 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.565529 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwgf\" (UniqueName: \"kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf\") pod \"authorino-5dc7ccb5f-kcfbl\" (UID: \"8c2685b1-f576-44f8-8ba7-6f7e1997d424\") " pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.581784 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.581761 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79b6bc8b5b-snzxn"] Sep 08 02:06:54.586121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.586105 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.590691 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.590639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b6bc8b5b-snzxn"] Sep 08 02:06:54.625108 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.625083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79b6bc8b5b-snzxn"] Sep 08 02:06:54.625380 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:06:54.625353 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vvtlb], unattached volumes=[], failed to process volumes=[kube-api-access-vvtlb]: context canceled" pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" podUID="3030a2fd-4a2e-4ba3-961a-fe1bd6919201" Sep 08 02:06:54.651911 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.651885 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:06:54.656320 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.656301 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.658422 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.658403 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 02:06:54.664006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.663986 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:06:54.757131 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.757100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.757270 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.757157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtlb\" (UniqueName: \"kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb\") pod \"authorino-79b6bc8b5b-snzxn\" (UID: \"3030a2fd-4a2e-4ba3-961a-fe1bd6919201\") " pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.757270 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.757205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffxs\" (UniqueName: \"kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.843015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.842955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.843344 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.842957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.850609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.850592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.853508 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.853494 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:54.857592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.857569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.857684 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.857629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtlb\" (UniqueName: \"kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb\") pod \"authorino-79b6bc8b5b-snzxn\" (UID: \"3030a2fd-4a2e-4ba3-961a-fe1bd6919201\") " pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.857684 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.857672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffxs\" (UniqueName: \"kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.859995 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.859974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.865455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.865438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtlb\" (UniqueName: \"kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb\") pod \"authorino-79b6bc8b5b-snzxn\" (UID: \"3030a2fd-4a2e-4ba3-961a-fe1bd6919201\") " pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:54.865781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.865763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffxs\" (UniqueName: \"kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs\") pod \"authorino-55bd44f5d6-r94p9\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:54.958387 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.958365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srwgf\" (UniqueName: \"kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf\") pod \"8c2685b1-f576-44f8-8ba7-6f7e1997d424\" (UID: \"8c2685b1-f576-44f8-8ba7-6f7e1997d424\") " Sep 08 02:06:54.960283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.960252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf" (OuterVolumeSpecName: "kube-api-access-srwgf") pod "8c2685b1-f576-44f8-8ba7-6f7e1997d424" (UID: "8c2685b1-f576-44f8-8ba7-6f7e1997d424"). InnerVolumeSpecName "kube-api-access-srwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:06:54.965366 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:54.965352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:06:55.059772 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.059739 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvtlb\" (UniqueName: \"kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb\") pod \"3030a2fd-4a2e-4ba3-961a-fe1bd6919201\" (UID: \"3030a2fd-4a2e-4ba3-961a-fe1bd6919201\") " Sep 08 02:06:55.059924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.059912 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwgf\" (UniqueName: \"kubernetes.io/projected/8c2685b1-f576-44f8-8ba7-6f7e1997d424-kube-api-access-srwgf\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:55.061689 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.061666 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb" (OuterVolumeSpecName: "kube-api-access-vvtlb") pod "3030a2fd-4a2e-4ba3-961a-fe1bd6919201" (UID: "3030a2fd-4a2e-4ba3-961a-fe1bd6919201"). InnerVolumeSpecName "kube-api-access-vvtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:06:55.160308 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.160268 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtlb\" (UniqueName: \"kubernetes.io/projected/3030a2fd-4a2e-4ba3-961a-fe1bd6919201-kube-api-access-vvtlb\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:06:55.233424 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.233400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:06:55.235448 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:06:55.235423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7af3d28_2cd5_4c19_bd13_3563839b060d.slice/crio-959151ad4b6afb4fe3c93a746780c7eaf8323ccb6ea9271d002a43e48d6d1ea0 WatchSource:0}: Error finding container 959151ad4b6afb4fe3c93a746780c7eaf8323ccb6ea9271d002a43e48d6d1ea0: Status 404 returned error can't find the container with id 959151ad4b6afb4fe3c93a746780c7eaf8323ccb6ea9271d002a43e48d6d1ea0 Sep 08 02:06:55.850753 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.850675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" event={"ID":"d7af3d28-2cd5-4c19-bd13-3563839b060d","Type":"ContainerStarted","Data":"959151ad4b6afb4fe3c93a746780c7eaf8323ccb6ea9271d002a43e48d6d1ea0"} Sep 08 02:06:55.851076 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.850746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6bc8b5b-snzxn" Sep 08 02:06:55.851076 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.850746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kcfbl" Sep 08 02:06:55.876940 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.876912 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kcfbl"] Sep 08 02:06:55.882097 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.882074 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kcfbl"] Sep 08 02:06:55.897777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.897753 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79b6bc8b5b-snzxn"] Sep 08 02:06:55.903823 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:55.903797 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79b6bc8b5b-snzxn"] Sep 08 02:06:57.319037 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:57.319002 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3030a2fd-4a2e-4ba3-961a-fe1bd6919201" path="/var/lib/kubelet/pods/3030a2fd-4a2e-4ba3-961a-fe1bd6919201/volumes" Sep 08 02:06:57.319550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:06:57.319345 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2685b1-f576-44f8-8ba7-6f7e1997d424" path="/var/lib/kubelet/pods/8c2685b1-f576-44f8-8ba7-6f7e1997d424/volumes" Sep 08 02:07:02.891991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:02.891952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" event={"ID":"d7af3d28-2cd5-4c19-bd13-3563839b060d","Type":"ContainerStarted","Data":"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f"} Sep 08 02:07:02.906816 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:02.906770 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" podStartSLOduration=2.300939545 podStartE2EDuration="8.906755444s" totalImagesPullingTime="6.605815899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:02.905807166 +0000 UTC m=+504.083119247" watchObservedRunningTime="2026-09-08 02:07:02.906755444 +0000 UTC m=+504.084067516" Sep 08 02:07:02.931711 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:02.931680 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:07:02.931891 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:02.931836 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" podUID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" containerName="authorino" containerID="cri-o://7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6" gracePeriod=30 Sep 08 02:07:03.178886 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.178854 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:07:03.220526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.220496 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdbmt\" (UniqueName: \"kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt\") pod \"e4a5ae43-b8e7-4627-a868-7f42b2a79404\" (UID: \"e4a5ae43-b8e7-4627-a868-7f42b2a79404\") " Sep 08 02:07:03.222931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.222902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt" (OuterVolumeSpecName: "kube-api-access-wdbmt") pod "e4a5ae43-b8e7-4627-a868-7f42b2a79404" (UID: "e4a5ae43-b8e7-4627-a868-7f42b2a79404"). InnerVolumeSpecName "kube-api-access-wdbmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:07:03.321775 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.321723 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbmt\" (UniqueName: \"kubernetes.io/projected/e4a5ae43-b8e7-4627-a868-7f42b2a79404-kube-api-access-wdbmt\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:07:03.899371 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.899334 2559 generic.go:356] "Generic (PLEG): container finished" podID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" containerID="7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6" exitCode=0 Sep 08 02:07:03.899810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.899387 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" Sep 08 02:07:03.899810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.899402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" event={"ID":"e4a5ae43-b8e7-4627-a868-7f42b2a79404","Type":"ContainerDied","Data":"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6"} Sep 08 02:07:03.899810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.899446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-nj7mn" event={"ID":"e4a5ae43-b8e7-4627-a868-7f42b2a79404","Type":"ContainerDied","Data":"0488d34a7f85f713e4e8b1ae5ae7d6c27ad590a2111183e5e86ea7d888d1818d"} Sep 08 02:07:03.899810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.899465 2559 scope.go:117] "RemoveContainer" containerID="7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6" Sep 08 02:07:03.908906 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.908887 2559 scope.go:117] "RemoveContainer" containerID="7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6" Sep 08 02:07:03.909181 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:07:03.909165 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6\": container with ID starting with 7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6 not found: ID does not exist" containerID="7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6" Sep 08 02:07:03.909227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.909188 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6"} err="failed to get container status \"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6\": rpc error: code = NotFound desc = could not find container \"7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6\": container with ID starting with 7046b1a32cdf19e747e9569c789d57e7205ffb076328aafb3a9f8ddc055914c6 not found: ID does not exist" Sep 08 02:07:03.914503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.914481 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:07:03.918482 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:03.918462 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-nj7mn"] Sep 08 02:07:05.320134 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:05.320101 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" path="/var/lib/kubelet/pods/e4a5ae43-b8e7-4627-a868-7f42b2a79404/volumes" Sep 08 02:07:07.677231 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.677199 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:07:07.677747 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.677730 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" containerName="authorino" Sep 08 02:07:07.677747 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.677749 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" containerName="authorino" Sep 08 02:07:07.677842 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.677816 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a5ae43-b8e7-4627-a868-7f42b2a79404" containerName="authorino" Sep 08 02:07:07.682114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.682096 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.684210 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.684184 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 02:07:07.684353 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.684278 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 02:07:07.685051 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.685036 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc7lc\"" Sep 08 02:07:07.685142 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.685039 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 02:07:07.691454 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.691431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:07:07.759463 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.759414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkq5\" (UniqueName: \"kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.759677 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.759468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.759677 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.759518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.860429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.860396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.860429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.860429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.860804 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.860509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkq5\" (UniqueName: \"kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.860804 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:07:07.860600 2559 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 08 02:07:07.860804 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:07:07.860724 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert podName:1f7db392-58ce-4167-bef6-d9c138984d97 nodeName:}" failed. No retries permitted until 2026-09-08 02:07:08.360698096 +0000 UTC m=+509.538010160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert") pod "maas-controller-5694fc4746-tfdtk" (UID: "1f7db392-58ce-4167-bef6-d9c138984d97") : secret "maas-controller-webhook-cert" not found Sep 08 02:07:07.863072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.863046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:07.871380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:07.871356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkq5\" (UniqueName: \"kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:08.364645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:08.364605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:08.366996 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:08.366971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") pod \"maas-controller-5694fc4746-tfdtk\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:08.594285 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:08.594243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:08.866783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:08.866758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:07:08.868420 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:07:08.868392 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7db392_58ce_4167_bef6_d9c138984d97.slice/crio-81849ac1591120ecfc56cbcc5015ae8e6e24bc554d961e5ae4bbb2ff5ac9b1b2 WatchSource:0}: Error finding container 81849ac1591120ecfc56cbcc5015ae8e6e24bc554d961e5ae4bbb2ff5ac9b1b2: Status 404 returned error can't find the container with id 81849ac1591120ecfc56cbcc5015ae8e6e24bc554d961e5ae4bbb2ff5ac9b1b2 Sep 08 02:07:08.927654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:08.927613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-tfdtk" event={"ID":"1f7db392-58ce-4167-bef6-d9c138984d97","Type":"ContainerStarted","Data":"81849ac1591120ecfc56cbcc5015ae8e6e24bc554d961e5ae4bbb2ff5ac9b1b2"} Sep 08 02:07:11.948202 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:11.948167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-tfdtk" event={"ID":"1f7db392-58ce-4167-bef6-d9c138984d97","Type":"ContainerStarted","Data":"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5"} Sep 08 02:07:11.948621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:11.948318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:11.966363 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:11.966317 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5694fc4746-tfdtk" podStartSLOduration=2.432128504 podStartE2EDuration="4.966304155s" totalImagesPullingTime="2.534175651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:11.963979202 +0000 UTC m=+513.141291274" watchObservedRunningTime="2026-09-08 02:07:11.966304155 +0000 UTC m=+513.143616267" Sep 08 02:07:22.955761 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:22.955729 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:07:45.454958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.454923 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:07:45.461266 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.461246 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.464347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.464326 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 02:07:45.464694 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.464326 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tfvqz\"" Sep 08 02:07:45.464694 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.464391 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 02:07:45.471137 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.471117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:07:45.572564 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.572528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqn4\" (UniqueName: \"kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.572735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.572587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.572735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.572624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.572735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.572666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.673820 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.673780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.673998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.673830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.673998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.673890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.673998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.673952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqn4\" (UniqueName: \"kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.674741 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.674711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.676415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.676389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.676532 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.676442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.681744 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.681716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqn4\" (UniqueName: \"kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4\") pod \"maas-api-7df9d55888-dwb8z\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.773949 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.773855 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:45.951415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.951383 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-klkkd"] Sep 08 02:07:45.955010 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.954995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:45.957114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.957092 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9n5m5\"" Sep 08 02:07:45.957245 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.957124 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 02:07:45.962523 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:45.962498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-klkkd"] Sep 08 02:07:46.053913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.053871 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:07:46.057062 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:07:46.057032 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0778aebe_bf73_40c3_a937_2e5efe46384e.slice/crio-26643b12ae73f4ace6414176b728a542a3ace2e3d583bb7140d2c2b58e4e44ab WatchSource:0}: Error finding container 26643b12ae73f4ace6414176b728a542a3ace2e3d583bb7140d2c2b58e4e44ab: Status 404 returned error can't find the container with id 26643b12ae73f4ace6414176b728a542a3ace2e3d583bb7140d2c2b58e4e44ab Sep 08 02:07:46.077474 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.077436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd254\" (UniqueName: \"kubernetes.io/projected/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-kube-api-access-xd254\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.077622 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.077535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.138524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.138490 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" event={"ID":"0778aebe-bf73-40c3-a937-2e5efe46384e","Type":"ContainerStarted","Data":"26643b12ae73f4ace6414176b728a542a3ace2e3d583bb7140d2c2b58e4e44ab"} Sep 08 02:07:46.178016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.177980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd254\" (UniqueName: \"kubernetes.io/projected/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-kube-api-access-xd254\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.178192 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.178048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.178772 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.178753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.188351 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.188319 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd254\" (UniqueName: \"kubernetes.io/projected/df82d1d3-9c8e-4fc2-95c2-09655a1f5b41-kube-api-access-xd254\") pod \"payload-processing-5b54b7fbb7-klkkd\" (UID: \"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.267459 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.267420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:46.543832 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:46.543810 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-klkkd"] Sep 08 02:07:46.545627 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:07:46.545603 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf82d1d3_9c8e_4fc2_95c2_09655a1f5b41.slice/crio-f607ab23db439923bf8864c15294c40a5d7a93861b3434072cc3edff60933664 WatchSource:0}: Error finding container f607ab23db439923bf8864c15294c40a5d7a93861b3434072cc3edff60933664: Status 404 returned error can't find the container with id f607ab23db439923bf8864c15294c40a5d7a93861b3434072cc3edff60933664 Sep 08 02:07:47.148239 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:47.148159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" event={"ID":"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41","Type":"ContainerStarted","Data":"f607ab23db439923bf8864c15294c40a5d7a93861b3434072cc3edff60933664"} Sep 08 02:07:48.167535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:48.167488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" event={"ID":"0778aebe-bf73-40c3-a937-2e5efe46384e","Type":"ContainerStarted","Data":"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc"} Sep 08 02:07:48.167959 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:48.167736 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:07:48.183974 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:48.183928 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" podStartSLOduration=1.399549554 podStartE2EDuration="3.183912335s" totalImagesPullingTime="1.784362781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:48.182422707 +0000 UTC m=+549.359734778" watchObservedRunningTime="2026-09-08 02:07:48.183912335 +0000 UTC m=+549.361224406" Sep 08 02:07:49.175071 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:49.175025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" event={"ID":"df82d1d3-9c8e-4fc2-95c2-09655a1f5b41","Type":"ContainerStarted","Data":"9c7b9043d1623c37a319bce1b63d6bf26d2cf932482e47ac58b7c628ff116725"} Sep 08 02:07:49.175599 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:49.175267 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:07:49.189683 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:49.189636 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" podStartSLOduration=2.409968886 podStartE2EDuration="4.189621951s" totalImagesPullingTime="1.779653065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:49.188585151 +0000 UTC m=+550.365897225" watchObservedRunningTime="2026-09-08 02:07:49.189621951 +0000 UTC m=+550.366934023" Sep 08 02:07:54.180581 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:07:54.180548 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:08:00.182934 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:00.182903 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-klkkd" Sep 08 02:08:31.212662 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.212618 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb"] Sep 08 02:08:31.219313 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.219268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.222669 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.222647 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x2jnv\"" Sep 08 02:08:31.222870 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.222849 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 08 02:08:31.223021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.222715 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 02:08:31.223198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.222744 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 02:08:31.226839 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.226814 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb"] Sep 08 02:08:31.373244 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxg9\" (UniqueName: \"kubernetes.io/projected/53246d60-6795-4095-8e78-d0411e6642ea-kube-api-access-pdxg9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.373449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.373449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.373449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.373449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.373449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.373445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53246d60-6795-4095-8e78-d0411e6642ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.473938 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.473858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxg9\" (UniqueName: \"kubernetes.io/projected/53246d60-6795-4095-8e78-d0411e6642ea-kube-api-access-pdxg9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.473938 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.473896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.473938 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.473917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.473946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.473990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.474056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53246d60-6795-4095-8e78-d0411e6642ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.474361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.474387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.474548 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.474469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.476423 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.476400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53246d60-6795-4095-8e78-d0411e6642ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.476704 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.476687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53246d60-6795-4095-8e78-d0411e6642ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.482509 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.482484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxg9\" (UniqueName: \"kubernetes.io/projected/53246d60-6795-4095-8e78-d0411e6642ea-kube-api-access-pdxg9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9kbb\" (UID: \"53246d60-6795-4095-8e78-d0411e6642ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.531232 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.531188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:31.805740 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:31.805704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb"] Sep 08 02:08:31.806930 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:08:31.806904 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53246d60_6795_4095_8e78_d0411e6642ea.slice/crio-dd8847cafcb95c000baf7cd2fe47b71f14b6d4bdb3aa92d35d986bbbf002b033 WatchSource:0}: Error finding container dd8847cafcb95c000baf7cd2fe47b71f14b6d4bdb3aa92d35d986bbbf002b033: Status 404 returned error can't find the container with id dd8847cafcb95c000baf7cd2fe47b71f14b6d4bdb3aa92d35d986bbbf002b033 Sep 08 02:08:32.419490 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:32.419455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" event={"ID":"53246d60-6795-4095-8e78-d0411e6642ea","Type":"ContainerStarted","Data":"dd8847cafcb95c000baf7cd2fe47b71f14b6d4bdb3aa92d35d986bbbf002b033"} Sep 08 02:08:33.426694 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:33.426655 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" event={"ID":"53246d60-6795-4095-8e78-d0411e6642ea","Type":"ContainerStarted","Data":"b05307d2441126a00ca7920bdd10a787d4ff0d7ee2c93e4b97178d81576a2e56"} Sep 08 02:08:33.444227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:33.444182 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" podStartSLOduration=1.441728018 podStartE2EDuration="2.44416293s" totalImagesPullingTime="1.002434912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:08:33.443908611 +0000 UTC m=+594.621220684" watchObservedRunningTime="2026-09-08 02:08:33.44416293 +0000 UTC m=+594.621475003" Sep 08 02:08:39.302999 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:39.302967 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:08:39.305189 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:39.305165 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:08:41.532226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:41.532192 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:41.532595 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:41.532257 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:41.544659 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:41.544635 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:08:42.486192 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:08:42.486162 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9kbb" Sep 08 02:10:58.199645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.199558 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67df555867-b7kqc"] Sep 08 02:10:58.203052 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.203032 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.210074 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.210047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67df555867-b7kqc"] Sep 08 02:10:58.348779 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.348746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md78n\" (UniqueName: \"kubernetes.io/projected/3a9c58fd-2883-4de6-834c-1f5e499addd4-kube-api-access-md78n\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.348960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.348818 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a9c58fd-2883-4de6-834c-1f5e499addd4-tls-cert\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.450168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.450085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a9c58fd-2883-4de6-834c-1f5e499addd4-tls-cert\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.450168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.450161 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md78n\" (UniqueName: \"kubernetes.io/projected/3a9c58fd-2883-4de6-834c-1f5e499addd4-kube-api-access-md78n\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.452620 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.452597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a9c58fd-2883-4de6-834c-1f5e499addd4-tls-cert\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.457629 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.457608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md78n\" (UniqueName: \"kubernetes.io/projected/3a9c58fd-2883-4de6-834c-1f5e499addd4-kube-api-access-md78n\") pod \"authorino-67df555867-b7kqc\" (UID: \"3a9c58fd-2883-4de6-834c-1f5e499addd4\") " pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.513686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.513655 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67df555867-b7kqc" Sep 08 02:10:58.785003 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.784976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67df555867-b7kqc"] Sep 08 02:10:58.787162 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:10:58.787134 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a9c58fd_2883_4de6_834c_1f5e499addd4.slice/crio-99c2f4b39e32c71d396b8396c9d236b9b8c3a28fc6d6709d7a9ab3efde8244c5 WatchSource:0}: Error finding container 99c2f4b39e32c71d396b8396c9d236b9b8c3a28fc6d6709d7a9ab3efde8244c5: Status 404 returned error can't find the container with id 99c2f4b39e32c71d396b8396c9d236b9b8c3a28fc6d6709d7a9ab3efde8244c5 Sep 08 02:10:58.788561 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:58.788542 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 02:10:59.246571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:10:59.246538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67df555867-b7kqc" event={"ID":"3a9c58fd-2883-4de6-834c-1f5e499addd4","Type":"ContainerStarted","Data":"99c2f4b39e32c71d396b8396c9d236b9b8c3a28fc6d6709d7a9ab3efde8244c5"} Sep 08 02:11:00.254042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.254001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67df555867-b7kqc" event={"ID":"3a9c58fd-2883-4de6-834c-1f5e499addd4","Type":"ContainerStarted","Data":"7dfc9619ddf97401e90a8809cab97b8fa1cd633631e790602d6c9b52581048ca"} Sep 08 02:11:00.274028 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.273970 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67df555867-b7kqc" podStartSLOduration=1.475776382 podStartE2EDuration="2.273953279s" totalImagesPullingTime="798.176897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:00.269796566 +0000 UTC m=+741.447108640" watchObservedRunningTime="2026-09-08 02:11:00.273953279 +0000 UTC m=+741.451265371" Sep 08 02:11:00.296957 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.296924 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:11:00.297162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.297120 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" podUID="d7af3d28-2cd5-4c19-bd13-3563839b060d" containerName="authorino" containerID="cri-o://48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f" gracePeriod=30 Sep 08 02:11:00.564692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.564660 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:11:00.670593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.670556 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert\") pod \"d7af3d28-2cd5-4c19-bd13-3563839b060d\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " Sep 08 02:11:00.670761 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.670622 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ffxs\" (UniqueName: \"kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs\") pod \"d7af3d28-2cd5-4c19-bd13-3563839b060d\" (UID: \"d7af3d28-2cd5-4c19-bd13-3563839b060d\") " Sep 08 02:11:00.672648 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.672618 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs" (OuterVolumeSpecName: "kube-api-access-2ffxs") pod "d7af3d28-2cd5-4c19-bd13-3563839b060d" (UID: "d7af3d28-2cd5-4c19-bd13-3563839b060d"). InnerVolumeSpecName "kube-api-access-2ffxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:11:00.682897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.682867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d7af3d28-2cd5-4c19-bd13-3563839b060d" (UID: "d7af3d28-2cd5-4c19-bd13-3563839b060d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:11:00.772321 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.772257 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7af3d28-2cd5-4c19-bd13-3563839b060d-tls-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:11:00.772321 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:00.772304 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffxs\" (UniqueName: \"kubernetes.io/projected/d7af3d28-2cd5-4c19-bd13-3563839b060d-kube-api-access-2ffxs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:11:01.262072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.262031 2559 generic.go:356] "Generic (PLEG): container finished" podID="d7af3d28-2cd5-4c19-bd13-3563839b060d" containerID="48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f" exitCode=0 Sep 08 02:11:01.262072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.262072 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" Sep 08 02:11:01.262550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.262107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" event={"ID":"d7af3d28-2cd5-4c19-bd13-3563839b060d","Type":"ContainerDied","Data":"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f"} Sep 08 02:11:01.262550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.262153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bd44f5d6-r94p9" event={"ID":"d7af3d28-2cd5-4c19-bd13-3563839b060d","Type":"ContainerDied","Data":"959151ad4b6afb4fe3c93a746780c7eaf8323ccb6ea9271d002a43e48d6d1ea0"} Sep 08 02:11:01.262550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.262176 2559 scope.go:117] "RemoveContainer" containerID="48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f" Sep 08 02:11:01.274020 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.273994 2559 scope.go:117] "RemoveContainer" containerID="48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f" Sep 08 02:11:01.274441 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:01.274414 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f\": container with ID starting with 48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f not found: ID does not exist" containerID="48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f" Sep 08 02:11:01.274642 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.274452 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f"} err="failed to get container status \"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f\": rpc error: code = NotFound desc = could not find container \"48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f\": container with ID starting with 48c846e71acba77af967d0082e2af9a16c59c1aa34727b15c6c3a5365cb1220f not found: ID does not exist" Sep 08 02:11:01.295025 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.294985 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:11:01.304336 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.304276 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55bd44f5d6-r94p9"] Sep 08 02:11:01.319712 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:01.319678 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7af3d28-2cd5-4c19-bd13-3563839b060d" path="/var/lib/kubelet/pods/d7af3d28-2cd5-4c19-bd13-3563839b060d/volumes" Sep 08 02:11:19.552750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.552715 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:19.553177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.553103 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7af3d28-2cd5-4c19-bd13-3563839b060d" containerName="authorino" Sep 08 02:11:19.553177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.553115 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7af3d28-2cd5-4c19-bd13-3563839b060d" containerName="authorino" Sep 08 02:11:19.553177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.553178 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7af3d28-2cd5-4c19-bd13-3563839b060d" containerName="authorino" Sep 08 02:11:19.556534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.556515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:19.558972 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.558945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2574n\"" Sep 08 02:11:19.560975 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.560953 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:19.746446 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.746406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtls\" (UniqueName: \"kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls\") pod \"test-curl-7691-140173147941680\" (UID: \"94300c3d-4867-4ffe-abda-cb8a1a19097d\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:19.847125 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.847043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtls\" (UniqueName: \"kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls\") pod \"test-curl-7691-140173147941680\" (UID: \"94300c3d-4867-4ffe-abda-cb8a1a19097d\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:19.860960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.860928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtls\" (UniqueName: \"kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls\") pod \"test-curl-7691-140173147941680\" (UID: \"94300c3d-4867-4ffe-abda-cb8a1a19097d\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:19.868659 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:19.868635 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:20.140319 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:20.140236 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:20.142795 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:20.142736 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94300c3d_4867_4ffe_abda_cb8a1a19097d.slice/crio-91d6cf332b0344e85df3ae34a0f9f938e1df1bf209624d991668f83155dd8459 WatchSource:0}: Error finding container 91d6cf332b0344e85df3ae34a0f9f938e1df1bf209624d991668f83155dd8459: Status 404 returned error can't find the container with id 91d6cf332b0344e85df3ae34a0f9f938e1df1bf209624d991668f83155dd8459 Sep 08 02:11:20.378045 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:20.378010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-140173147941680" event={"ID":"94300c3d-4867-4ffe-abda-cb8a1a19097d","Type":"ContainerStarted","Data":"91d6cf332b0344e85df3ae34a0f9f938e1df1bf209624d991668f83155dd8459"} Sep 08 02:11:21.384892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.384862 2559 generic.go:356] "Generic (PLEG): container finished" podID="94300c3d-4867-4ffe-abda-cb8a1a19097d" containerID="9f748a6887af20b7ed5ac4d70a7c8bb03f7880cebb43b7f9df9019ffc99f16dc" exitCode=0 Sep 08 02:11:21.385252 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.384921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-140173147941680" event={"ID":"94300c3d-4867-4ffe-abda-cb8a1a19097d","Type":"ContainerDied","Data":"9f748a6887af20b7ed5ac4d70a7c8bb03f7880cebb43b7f9df9019ffc99f16dc"} Sep 08 02:11:21.867758 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.867725 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:11:21.871162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.871137 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:21.873370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.873342 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5d27de-serving-cert\"" Sep 08 02:11:21.881354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:21.881334 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:11:22.067571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.067532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvmz\" (UniqueName: \"kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.067779 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.067600 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.067779 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.067643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.067887 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.067766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.168202 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.168169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.168366 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.168233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvmz\" (UniqueName: \"kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.168366 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.168276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.168366 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:22.168318 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-serving-cert: secret "maas-api-e2e-inf-a-5d27de-serving-cert" not found Sep 08 02:11:22.168515 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:22.168394 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls podName:91a67d75-3523-4125-9550-dc0f65a2ed51 nodeName:}" failed. No retries permitted until 2026-09-08 02:11:22.668369367 +0000 UTC m=+763.845681440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls") pod "maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" (UID: "91a67d75-3523-4125-9550-dc0f65a2ed51") : secret "maas-api-e2e-inf-a-5d27de-serving-cert" not found Sep 08 02:11:22.168515 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.168325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.168890 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.168864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.170833 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.170807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.176670 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.176647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvmz\" (UniqueName: \"kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.535518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.535482 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:22.571608 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.571550 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggtls\" (UniqueName: \"kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls\") pod \"94300c3d-4867-4ffe-abda-cb8a1a19097d\" (UID: \"94300c3d-4867-4ffe-abda-cb8a1a19097d\") " Sep 08 02:11:22.574493 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.574450 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls" (OuterVolumeSpecName: "kube-api-access-ggtls") pod "94300c3d-4867-4ffe-abda-cb8a1a19097d" (UID: "94300c3d-4867-4ffe-abda-cb8a1a19097d"). InnerVolumeSpecName "kube-api-access-ggtls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:11:22.577239 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.577199 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7691-140173147941680_94300c3d-4867-4ffe-abda-cb8a1a19097d/test-curl-7691-140173147941680/0.log" Sep 08 02:11:22.602197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.602159 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:22.605146 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.605122 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:22.672958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.672925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.673117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.672985 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtls\" (UniqueName: \"kubernetes.io/projected/94300c3d-4867-4ffe-abda-cb8a1a19097d-kube-api-access-ggtls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:11:22.675362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.675339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.722827 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.722751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:22.723188 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.723174 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94300c3d-4867-4ffe-abda-cb8a1a19097d" containerName="test-curl-7691-140173147941680" Sep 08 02:11:22.723236 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.723189 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="94300c3d-4867-4ffe-abda-cb8a1a19097d" containerName="test-curl-7691-140173147941680" Sep 08 02:11:22.723275 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.723265 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="94300c3d-4867-4ffe-abda-cb8a1a19097d" containerName="test-curl-7691-140173147941680" Sep 08 02:11:22.726763 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.726739 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:22.733556 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.733526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:22.773650 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.773614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jtl\" (UniqueName: \"kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl\") pod \"test-curl-7691-140173147941680\" (UID: \"465d0864-90cd-4de7-9ddf-5bd177771ead\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:22.783605 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.783569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:22.874099 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.874065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jtl\" (UniqueName: \"kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl\") pod \"test-curl-7691-140173147941680\" (UID: \"465d0864-90cd-4de7-9ddf-5bd177771ead\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:22.884006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:22.883977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jtl\" (UniqueName: \"kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl\") pod \"test-curl-7691-140173147941680\" (UID: \"465d0864-90cd-4de7-9ddf-5bd177771ead\") " pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:23.038646 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.038566 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:23.060783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.060737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:11:23.062027 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:23.062001 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a67d75_3523_4125_9550_dc0f65a2ed51.slice/crio-d89658d550d182e62cd232a3e89b1f947e86090c9ac8efd42d16815552de0625 WatchSource:0}: Error finding container d89658d550d182e62cd232a3e89b1f947e86090c9ac8efd42d16815552de0625: Status 404 returned error can't find the container with id d89658d550d182e62cd232a3e89b1f947e86090c9ac8efd42d16815552de0625 Sep 08 02:11:23.320718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.320693 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94300c3d-4867-4ffe-abda-cb8a1a19097d" path="/var/lib/kubelet/pods/94300c3d-4867-4ffe-abda-cb8a1a19097d/volumes" Sep 08 02:11:23.321795 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.321774 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:23.322715 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:23.322686 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod465d0864_90cd_4de7_9ddf_5bd177771ead.slice/crio-68709686bbde1e933d192fa4f7b7c5f84a32ab0cefaec422fa8c4194f787870a WatchSource:0}: Error finding container 68709686bbde1e933d192fa4f7b7c5f84a32ab0cefaec422fa8c4194f787870a: Status 404 returned error can't find the container with id 68709686bbde1e933d192fa4f7b7c5f84a32ab0cefaec422fa8c4194f787870a Sep 08 02:11:23.400039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.400009 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:23.400213 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.400011 2559 scope.go:117] "RemoveContainer" containerID="9f748a6887af20b7ed5ac4d70a7c8bb03f7880cebb43b7f9df9019ffc99f16dc" Sep 08 02:11:23.401515 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.401318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" event={"ID":"91a67d75-3523-4125-9550-dc0f65a2ed51","Type":"ContainerStarted","Data":"d89658d550d182e62cd232a3e89b1f947e86090c9ac8efd42d16815552de0625"} Sep 08 02:11:23.403013 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:23.402991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-140173147941680" event={"ID":"465d0864-90cd-4de7-9ddf-5bd177771ead","Type":"ContainerStarted","Data":"68709686bbde1e933d192fa4f7b7c5f84a32ab0cefaec422fa8c4194f787870a"} Sep 08 02:11:24.412712 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.412672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" event={"ID":"91a67d75-3523-4125-9550-dc0f65a2ed51","Type":"ContainerStarted","Data":"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec"} Sep 08 02:11:24.413129 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.412820 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:24.414392 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.414366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-140173147941680" event={"ID":"465d0864-90cd-4de7-9ddf-5bd177771ead","Type":"ContainerStarted","Data":"ab98e7eb886d329716d23cfa3716c6890f7357e5476396c2425796d8d0f9bc60"} Sep 08 02:11:24.434596 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.434547 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" podStartSLOduration=3.069112492 podStartE2EDuration="3.434528664s" totalImagesPullingTime="365.416172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:24.430879249 +0000 UTC m=+765.608191323" watchObservedRunningTime="2026-09-08 02:11:24.434528664 +0000 UTC m=+765.611840737" Sep 08 02:11:24.446462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.446413 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7691-140173147941680" podStartSLOduration=1.4571253419999999 podStartE2EDuration="2.446397647s" totalImagesPullingTime="989.272305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:24.44441724 +0000 UTC m=+765.621729314" watchObservedRunningTime="2026-09-08 02:11:24.446397647 +0000 UTC m=+765.623709720" Sep 08 02:11:24.525914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:24.525879 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7691-140173147941680_465d0864-90cd-4de7-9ddf-5bd177771ead/test-curl-7691-140173147941680/0.log" Sep 08 02:11:25.423012 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:25.422979 2559 generic.go:356] "Generic (PLEG): container finished" podID="465d0864-90cd-4de7-9ddf-5bd177771ead" containerID="ab98e7eb886d329716d23cfa3716c6890f7357e5476396c2425796d8d0f9bc60" exitCode=0 Sep 08 02:11:25.423423 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:25.423065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-140173147941680" event={"ID":"465d0864-90cd-4de7-9ddf-5bd177771ead","Type":"ContainerDied","Data":"ab98e7eb886d329716d23cfa3716c6890f7357e5476396c2425796d8d0f9bc60"} Sep 08 02:11:26.571283 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.571252 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:26.602867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.602834 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:26.608699 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.608663 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27jtl\" (UniqueName: \"kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl\") pod \"465d0864-90cd-4de7-9ddf-5bd177771ead\" (UID: \"465d0864-90cd-4de7-9ddf-5bd177771ead\") " Sep 08 02:11:26.609061 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.609041 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7691-140173147941680"] Sep 08 02:11:26.610743 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.610721 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl" (OuterVolumeSpecName: "kube-api-access-27jtl") pod "465d0864-90cd-4de7-9ddf-5bd177771ead" (UID: "465d0864-90cd-4de7-9ddf-5bd177771ead"). InnerVolumeSpecName "kube-api-access-27jtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:11:26.710279 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:26.710187 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jtl\" (UniqueName: \"kubernetes.io/projected/465d0864-90cd-4de7-9ddf-5bd177771ead-kube-api-access-27jtl\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:11:27.323158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:27.323112 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465d0864-90cd-4de7-9ddf-5bd177771ead" path="/var/lib/kubelet/pods/465d0864-90cd-4de7-9ddf-5bd177771ead/volumes" Sep 08 02:11:27.437085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:27.437051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-140173147941680" Sep 08 02:11:27.437085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:27.437065 2559 scope.go:117] "RemoveContainer" containerID="ab98e7eb886d329716d23cfa3716c6890f7357e5476396c2425796d8d0f9bc60" Sep 08 02:11:29.440678 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.440642 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:11:29.441588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.441559 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465d0864-90cd-4de7-9ddf-5bd177771ead" containerName="test-curl-7691-140173147941680" Sep 08 02:11:29.441756 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.441734 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="465d0864-90cd-4de7-9ddf-5bd177771ead" containerName="test-curl-7691-140173147941680" Sep 08 02:11:29.441917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.441905 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="465d0864-90cd-4de7-9ddf-5bd177771ead" containerName="test-curl-7691-140173147941680" Sep 08 02:11:29.446871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.446844 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.449979 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.449961 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-213d54df-serving-cert\"" Sep 08 02:11:29.464830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.459029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:11:29.532567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.532523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.532752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.532581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.532752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.532662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.532752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.532737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6hn\" (UniqueName: \"kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.633271 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.633233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6hn\" (UniqueName: \"kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.633669 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.633649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.633793 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.633778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.633932 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.633919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.634336 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:29.634310 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-serving-cert: secret "maas-api-e2e-ait-213d54df-serving-cert" not found Sep 08 02:11:29.634431 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:29.634397 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls podName:ce08fff3-015c-4e81-a52c-bf5b7e99cc1f nodeName:}" failed. No retries permitted until 2026-09-08 02:11:30.13436365 +0000 UTC m=+771.311675717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls") pod "maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" (UID: "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f") : secret "maas-api-e2e-ait-213d54df-serving-cert" not found Sep 08 02:11:29.634829 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.634806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.637481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.637443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:29.645736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:29.645710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6hn\" (UniqueName: \"kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:30.131846 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.131806 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:11:30.135716 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.135698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.138112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.138080 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-213d54df\"" Sep 08 02:11:30.138230 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.138187 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-213d54df-dockercfg-n9sj7\"" Sep 08 02:11:30.139222 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.139203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:30.141822 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.141802 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") pod \"maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:30.147567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.147546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:11:30.240833 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.240797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscpg\" (UniqueName: \"kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.241040 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.240851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.341771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.341731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscpg\" (UniqueName: \"kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.341962 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.341802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.342630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.342605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.348985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.348959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscpg\" (UniqueName: \"kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg\") pod \"payload-processing-e2e-ait-213d54df-86d569948f-mdn97\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.371773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.371741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:30.430498 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.430476 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:11:30.460710 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.460447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:30.650989 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.650965 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:11:30.653098 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:30.653072 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce08fff3_015c_4e81_a52c_bf5b7e99cc1f.slice/crio-70d5f9526970239034126ab7eebaf01ec78f07cd9f8c7ad964431464eca1661c WatchSource:0}: Error finding container 70d5f9526970239034126ab7eebaf01ec78f07cd9f8c7ad964431464eca1661c: Status 404 returned error can't find the container with id 70d5f9526970239034126ab7eebaf01ec78f07cd9f8c7ad964431464eca1661c Sep 08 02:11:30.737649 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:30.737623 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:11:30.738568 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:30.738543 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116377ef_eb4c_40a3_90dc_7476744498ac.slice/crio-9d7191a37febebfa772f0927f2ef1b91df87654fec1920269a2a4808c6b1bd79 WatchSource:0}: Error finding container 9d7191a37febebfa772f0927f2ef1b91df87654fec1920269a2a4808c6b1bd79: Status 404 returned error can't find the container with id 9d7191a37febebfa772f0927f2ef1b91df87654fec1920269a2a4808c6b1bd79 Sep 08 02:11:31.478258 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.478217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" event={"ID":"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f","Type":"ContainerStarted","Data":"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45"} Sep 08 02:11:31.478713 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.478276 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" event={"ID":"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f","Type":"ContainerStarted","Data":"70d5f9526970239034126ab7eebaf01ec78f07cd9f8c7ad964431464eca1661c"} Sep 08 02:11:31.478713 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.478332 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:31.479860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.479841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" event={"ID":"116377ef-eb4c-40a3-90dc-7476744498ac","Type":"ContainerStarted","Data":"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058"} Sep 08 02:11:31.479960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.479874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" event={"ID":"116377ef-eb4c-40a3-90dc-7476744498ac","Type":"ContainerStarted","Data":"9d7191a37febebfa772f0927f2ef1b91df87654fec1920269a2a4808c6b1bd79"} Sep 08 02:11:31.480019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.479973 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:31.498354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.498312 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" podStartSLOduration=2.115083933 podStartE2EDuration="2.498283717s" totalImagesPullingTime="383.199784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:31.497079552 +0000 UTC m=+772.674391635" watchObservedRunningTime="2026-09-08 02:11:31.498283717 +0000 UTC m=+772.675595788" Sep 08 02:11:31.513994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:31.513946 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" podStartSLOduration=1.5139323770000002 podStartE2EDuration="1.513932377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:31.512659358 +0000 UTC m=+772.689971430" watchObservedRunningTime="2026-09-08 02:11:31.513932377 +0000 UTC m=+772.691244506" Sep 08 02:11:32.206121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.206084 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:11:32.210384 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.210259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.212429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.212391 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-308137-serving-cert\"" Sep 08 02:11:32.218353 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.218328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:11:32.257525 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.257478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.257525 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.257522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.257768 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.257591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.257768 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.257642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.358238 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.358205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.358467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.358280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.358467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.358319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.358467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.358383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.358679 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:32.358656 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-308137-serving-cert: secret "maas-api-e2e-autores-308137-serving-cert" not found Sep 08 02:11:32.358737 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:11:32.358729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls podName:b648f20c-a699-41d1-bf4e-e9cb7f35c04f nodeName:}" failed. No retries permitted until 2026-09-08 02:11:32.858706806 +0000 UTC m=+774.036018874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls") pod "maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" (UID: "b648f20c-a699-41d1-bf4e-e9cb7f35c04f") : secret "maas-api-e2e-autores-308137-serving-cert" not found Sep 08 02:11:32.359334 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.359301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.360978 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.360952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.368090 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.368067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.863942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.863894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:32.866397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:32.866373 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") pod \"maas-api-e2e-autores-308137-556f6d8f8b-z6bjb\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:33.125187 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:33.125091 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:33.399439 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:33.399413 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:11:33.401947 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:33.401919 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb648f20c_a699_41d1_bf4e_e9cb7f35c04f.slice/crio-0cdf6ab5b143aa5384bdd9ed6eef875dbbd5710697e126d900284ae3af3ab314 WatchSource:0}: Error finding container 0cdf6ab5b143aa5384bdd9ed6eef875dbbd5710697e126d900284ae3af3ab314: Status 404 returned error can't find the container with id 0cdf6ab5b143aa5384bdd9ed6eef875dbbd5710697e126d900284ae3af3ab314 Sep 08 02:11:33.493631 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:33.493596 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" event={"ID":"b648f20c-a699-41d1-bf4e-e9cb7f35c04f","Type":"ContainerStarted","Data":"0cdf6ab5b143aa5384bdd9ed6eef875dbbd5710697e126d900284ae3af3ab314"} Sep 08 02:11:34.507885 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:34.507845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" event={"ID":"b648f20c-a699-41d1-bf4e-e9cb7f35c04f","Type":"ContainerStarted","Data":"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3"} Sep 08 02:11:34.508317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:34.508013 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:34.527006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:34.526949 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" podStartSLOduration=2.146768247 podStartE2EDuration="2.52693104s" totalImagesPullingTime="380.162793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:34.525677246 +0000 UTC m=+775.702989332" watchObservedRunningTime="2026-09-08 02:11:34.52693104 +0000 UTC m=+775.704243112" Sep 08 02:11:37.492550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:37.492518 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:11:40.519267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:40.519228 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:11:42.489085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:42.489056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:11:50.007758 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.007726 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:11:50.011918 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.011894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.014004 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.013977 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8d800a-serving-cert\"" Sep 08 02:11:50.023266 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.023237 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:11:50.133344 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.133282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.133537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.133414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.133537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.133466 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdkd\" (UniqueName: \"kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.133619 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.133568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.234396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.234360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.234396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.234400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.234619 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.234442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.234619 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.234462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdkd\" (UniqueName: \"kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.235200 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.235177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.236998 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.236966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.237114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.237029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.241582 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.241557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdkd\" (UniqueName: \"kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd\") pod \"maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.323964 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.323877 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:50.608120 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:50.608095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:11:50.610381 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:11:50.610324 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b364411_3dce_4b3a_a0f3_58dc6d726d8f.slice/crio-053359e159da9f477fba8f5ac228a7f6aef1ac035a9554bfba1a56199f104f90 WatchSource:0}: Error finding container 053359e159da9f477fba8f5ac228a7f6aef1ac035a9554bfba1a56199f104f90: Status 404 returned error can't find the container with id 053359e159da9f477fba8f5ac228a7f6aef1ac035a9554bfba1a56199f104f90 Sep 08 02:11:51.612544 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:51.612512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" event={"ID":"5b364411-3dce-4b3a-a0f3-58dc6d726d8f","Type":"ContainerStarted","Data":"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200"} Sep 08 02:11:51.612899 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:51.612576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" event={"ID":"5b364411-3dce-4b3a-a0f3-58dc6d726d8f","Type":"ContainerStarted","Data":"053359e159da9f477fba8f5ac228a7f6aef1ac035a9554bfba1a56199f104f90"} Sep 08 02:11:51.612899 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:51.612599 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:11:51.630006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:51.629928 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" podStartSLOduration=2.282621939 podStartE2EDuration="2.629903079s" totalImagesPullingTime="347.28114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:51.628657083 +0000 UTC m=+792.805969255" watchObservedRunningTime="2026-09-08 02:11:51.629903079 +0000 UTC m=+792.807215216" Sep 08 02:11:57.624483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:11:57.624447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:12:10.199024 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.198930 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:12:10.199460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.199161 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" podUID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" containerName="maas-api" containerID="cri-o://728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45" gracePeriod=30 Sep 08 02:12:10.256451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.256410 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:12:10.256744 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.256703 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" podUID="116377ef-eb4c-40a3-90dc-7476744498ac" containerName="payload-processing" containerID="cri-o://e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058" gracePeriod=30 Sep 08 02:12:10.518000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.517652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:12:10.520626 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.520599 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") pod \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " Sep 08 02:12:10.520745 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.520657 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs\") pod \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " Sep 08 02:12:10.520745 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.520687 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca\") pod \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " Sep 08 02:12:10.520745 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.520738 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm6hn\" (UniqueName: \"kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn\") pod \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\" (UID: \"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f\") " Sep 08 02:12:10.521250 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.521212 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" (UID: "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:12:10.525248 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.525210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" (UID: "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:12:10.527746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.527710 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" (UID: "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:12:10.529441 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.529395 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn" (OuterVolumeSpecName: "kube-api-access-dm6hn") pod "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" (UID: "ce08fff3-015c-4e81-a52c-bf5b7e99cc1f"). InnerVolumeSpecName "kube-api-access-dm6hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:12:10.562819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.562543 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:12:10.622335 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622306 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume\") pod \"116377ef-eb4c-40a3-90dc-7476744498ac\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " Sep 08 02:12:10.622516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622368 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hscpg\" (UniqueName: \"kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg\") pod \"116377ef-eb4c-40a3-90dc-7476744498ac\" (UID: \"116377ef-eb4c-40a3-90dc-7476744498ac\") " Sep 08 02:12:10.622652 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622522 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.622652 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622531 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.622652 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622540 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.622652 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622548 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6hn\" (UniqueName: \"kubernetes.io/projected/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f-kube-api-access-dm6hn\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.622829 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.622731 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "116377ef-eb4c-40a3-90dc-7476744498ac" (UID: "116377ef-eb4c-40a3-90dc-7476744498ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:12:10.624365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.624345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg" (OuterVolumeSpecName: "kube-api-access-hscpg") pod "116377ef-eb4c-40a3-90dc-7476744498ac" (UID: "116377ef-eb4c-40a3-90dc-7476744498ac"). InnerVolumeSpecName "kube-api-access-hscpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:12:10.723014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.722940 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/116377ef-eb4c-40a3-90dc-7476744498ac-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.723014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.722966 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscpg\" (UniqueName: \"kubernetes.io/projected/116377ef-eb4c-40a3-90dc-7476744498ac-kube-api-access-hscpg\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:10.729916 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.729884 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" containerID="728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45" exitCode=0 Sep 08 02:12:10.730069 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.729924 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" Sep 08 02:12:10.730069 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.729973 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" event={"ID":"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f","Type":"ContainerDied","Data":"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45"} Sep 08 02:12:10.730069 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.730033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb" event={"ID":"ce08fff3-015c-4e81-a52c-bf5b7e99cc1f","Type":"ContainerDied","Data":"70d5f9526970239034126ab7eebaf01ec78f07cd9f8c7ad964431464eca1661c"} Sep 08 02:12:10.730069 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.730063 2559 scope.go:117] "RemoveContainer" containerID="728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45" Sep 08 02:12:10.731860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.731834 2559 generic.go:356] "Generic (PLEG): container finished" podID="116377ef-eb4c-40a3-90dc-7476744498ac" containerID="e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058" exitCode=0 Sep 08 02:12:10.731976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.731870 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" Sep 08 02:12:10.731976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.731883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" event={"ID":"116377ef-eb4c-40a3-90dc-7476744498ac","Type":"ContainerDied","Data":"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058"} Sep 08 02:12:10.731976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.731923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97" event={"ID":"116377ef-eb4c-40a3-90dc-7476744498ac","Type":"ContainerDied","Data":"9d7191a37febebfa772f0927f2ef1b91df87654fec1920269a2a4808c6b1bd79"} Sep 08 02:12:10.740623 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.740602 2559 scope.go:117] "RemoveContainer" containerID="728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45" Sep 08 02:12:10.740880 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:10.740861 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45\": container with ID starting with 728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45 not found: ID does not exist" containerID="728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45" Sep 08 02:12:10.740957 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.740886 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45"} err="failed to get container status \"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45\": rpc error: code = NotFound desc = could not find container \"728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45\": container with ID starting with 728d8b53cb9dcee92b3c7d458168e967f1c871b21d8dcceb1ec04490bb9a1c45 not found: ID does not exist" Sep 08 02:12:10.740957 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.740903 2559 scope.go:117] "RemoveContainer" containerID="e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058" Sep 08 02:12:10.749032 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.749016 2559 scope.go:117] "RemoveContainer" containerID="e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058" Sep 08 02:12:10.749372 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:10.749329 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058\": container with ID starting with e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058 not found: ID does not exist" containerID="e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058" Sep 08 02:12:10.749469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.749380 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058"} err="failed to get container status \"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058\": rpc error: code = NotFound desc = could not find container \"e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058\": container with ID starting with e4bd77c3f94b822bd3ba59298831cdc9021557acb6708e287d4627c7560e5058 not found: ID does not exist" Sep 08 02:12:10.752831 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.752810 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:12:10.756020 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.756000 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-213d54df-8c4bdc9d-64cbb"] Sep 08 02:12:10.767856 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.767837 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:12:10.773173 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:10.773152 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-213d54df-86d569948f-mdn97"] Sep 08 02:12:11.320894 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:11.320861 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116377ef-eb4c-40a3-90dc-7476744498ac" path="/var/lib/kubelet/pods/116377ef-eb4c-40a3-90dc-7476744498ac/volumes" Sep 08 02:12:11.321258 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:11.321244 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" path="/var/lib/kubelet/pods/ce08fff3-015c-4e81-a52c-bf5b7e99cc1f/volumes" Sep 08 02:12:14.733033 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.732996 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:14.733691 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733674 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116377ef-eb4c-40a3-90dc-7476744498ac" containerName="payload-processing" Sep 08 02:12:14.733733 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733695 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="116377ef-eb4c-40a3-90dc-7476744498ac" containerName="payload-processing" Sep 08 02:12:14.733733 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733708 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" containerName="maas-api" Sep 08 02:12:14.733733 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733717 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" containerName="maas-api" Sep 08 02:12:14.733850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733838 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce08fff3-015c-4e81-a52c-bf5b7e99cc1f" containerName="maas-api" Sep 08 02:12:14.733887 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.733856 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="116377ef-eb4c-40a3-90dc-7476744498ac" containerName="payload-processing" Sep 08 02:12:14.739042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.739022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.741368 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.741350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ea979531-serving-cert\"" Sep 08 02:12:14.748463 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.748443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:14.756000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.755979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.756104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.756008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98z7\" (UniqueName: \"kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.756104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.756039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.756227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.756140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.857446 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.857408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.857621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.857522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.857621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.857551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98z7\" (UniqueName: \"kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.857621 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:14.857565 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-serving-cert: secret "maas-api-e2e-migrate-ea979531-serving-cert" not found Sep 08 02:12:14.857621 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.857608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.857826 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:14.857628 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls podName:0c2235d0-865e-4739-939b-b4d3b57a8409 nodeName:}" failed. No retries permitted until 2026-09-08 02:12:15.357606819 +0000 UTC m=+816.534918892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls") pod "maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" (UID: "0c2235d0-865e-4739-939b-b4d3b57a8409") : secret "maas-api-e2e-migrate-ea979531-serving-cert" not found Sep 08 02:12:14.858377 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.858349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.860247 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.860221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:14.871520 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:14.871490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98z7\" (UniqueName: \"kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:15.361425 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.361388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:15.363723 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.363693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") pod \"maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:15.595336 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.595304 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:15.599172 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.599155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.601775 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.601628 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ea979531-dockercfg-zstx8\"" Sep 08 02:12:15.601775 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.601677 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ea979531\"" Sep 08 02:12:15.607772 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.607702 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:15.650778 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.650739 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:15.664148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.664115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.664424 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.664405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnx2q\" (UniqueName: \"kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.685363 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.685333 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:15.689264 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.689243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.704163 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.704132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:15.766462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.765258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnx2q\" (UniqueName: \"kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.766462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.765329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcm5s\" (UniqueName: \"kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.766462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.765363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.766462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.765487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.766462 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.766455 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.774104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.774080 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnx2q\" (UniqueName: \"kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q\") pod \"payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.866793 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.866751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcm5s\" (UniqueName: \"kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.866987 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.866804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.867771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.867742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.877779 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.877754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcm5s\" (UniqueName: \"kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s\") pod \"payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:15.911686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.911595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:15.939664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:15.939637 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:15.941840 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:15.941816 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2235d0_865e_4739_939b_b4d3b57a8409.slice/crio-53efb8b9826885ca733a9e665aa304999d9e3df205d5f6a5c677747eaa326f4e WatchSource:0}: Error finding container 53efb8b9826885ca733a9e665aa304999d9e3df205d5f6a5c677747eaa326f4e: Status 404 returned error can't find the container with id 53efb8b9826885ca733a9e665aa304999d9e3df205d5f6a5c677747eaa326f4e Sep 08 02:12:16.001691 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.001661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:16.193067 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.193040 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:16.194677 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:16.194649 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9389835_5fbe_469b_a881_6fa52872fec6.slice/crio-6417a7135d8153d6608a196565e25851e9a6a932a6a74c3c6673a81a5e68cad8 WatchSource:0}: Error finding container 6417a7135d8153d6608a196565e25851e9a6a932a6a74c3c6673a81a5e68cad8: Status 404 returned error can't find the container with id 6417a7135d8153d6608a196565e25851e9a6a932a6a74c3c6673a81a5e68cad8 Sep 08 02:12:16.281248 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.281222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:16.282674 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:16.282649 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb78257_f484_43ba_b84b_61365804f332.slice/crio-07216b1f7b84951f95dacfe39b5e0da38717b09abf52e04edb330ed2866a3d0f WatchSource:0}: Error finding container 07216b1f7b84951f95dacfe39b5e0da38717b09abf52e04edb330ed2866a3d0f: Status 404 returned error can't find the container with id 07216b1f7b84951f95dacfe39b5e0da38717b09abf52e04edb330ed2866a3d0f Sep 08 02:12:16.790056 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.790018 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" event={"ID":"0c2235d0-865e-4739-939b-b4d3b57a8409","Type":"ContainerStarted","Data":"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4"} Sep 08 02:12:16.790495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.790106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" event={"ID":"0c2235d0-865e-4739-939b-b4d3b57a8409","Type":"ContainerStarted","Data":"53efb8b9826885ca733a9e665aa304999d9e3df205d5f6a5c677747eaa326f4e"} Sep 08 02:12:16.790495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.790135 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:16.792015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.791986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" event={"ID":"7bb78257-f484-43ba-b84b-61365804f332","Type":"ContainerStarted","Data":"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365"} Sep 08 02:12:16.792125 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.792041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:16.792125 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.792066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" event={"ID":"7bb78257-f484-43ba-b84b-61365804f332","Type":"ContainerStarted","Data":"07216b1f7b84951f95dacfe39b5e0da38717b09abf52e04edb330ed2866a3d0f"} Sep 08 02:12:16.793800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.793777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" event={"ID":"d9389835-5fbe-469b-a881-6fa52872fec6","Type":"ContainerStarted","Data":"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0"} Sep 08 02:12:16.793905 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.793824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" event={"ID":"d9389835-5fbe-469b-a881-6fa52872fec6","Type":"ContainerStarted","Data":"6417a7135d8153d6608a196565e25851e9a6a932a6a74c3c6673a81a5e68cad8"} Sep 08 02:12:16.793905 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.793876 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:16.808635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.808588 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" podStartSLOduration=2.502555729 podStartE2EDuration="2.808571744s" totalImagesPullingTime="306.016015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:16.805255916 +0000 UTC m=+817.982568001" watchObservedRunningTime="2026-09-08 02:12:16.808571744 +0000 UTC m=+817.985883813" Sep 08 02:12:16.820954 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.820905 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" podStartSLOduration=1.8208899029999999 podStartE2EDuration="1.820889903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:16.819363194 +0000 UTC m=+817.996675265" watchObservedRunningTime="2026-09-08 02:12:16.820889903 +0000 UTC m=+817.998202019" Sep 08 02:12:16.836798 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:16.836750 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" podStartSLOduration=1.836732839 podStartE2EDuration="1.836732839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:16.833239396 +0000 UTC m=+818.010551470" watchObservedRunningTime="2026-09-08 02:12:16.836732839 +0000 UTC m=+818.014044909" Sep 08 02:12:22.805632 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:22.805601 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:27.802013 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:27.801983 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:27.802505 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:27.802085 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:46.231989 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.231954 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:46.232558 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.232235 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" podUID="0c2235d0-865e-4739-939b-b4d3b57a8409" containerName="maas-api" containerID="cri-o://fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4" gracePeriod=30 Sep 08 02:12:46.287625 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.287593 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:46.287839 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.287807 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" podUID="7bb78257-f484-43ba-b84b-61365804f332" containerName="payload-processing" containerID="cri-o://5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365" gracePeriod=30 Sep 08 02:12:46.294519 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.294491 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:46.294807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.294745 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" podUID="d9389835-5fbe-469b-a881-6fa52872fec6" containerName="payload-pre-processing" containerID="cri-o://9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0" gracePeriod=30 Sep 08 02:12:46.570571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.570499 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:46.626475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.626452 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:46.629476 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.629457 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:46.648986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.648957 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcm5s\" (UniqueName: \"kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s\") pod \"7bb78257-f484-43ba-b84b-61365804f332\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " Sep 08 02:12:46.649116 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.649022 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume\") pod \"7bb78257-f484-43ba-b84b-61365804f332\" (UID: \"7bb78257-f484-43ba-b84b-61365804f332\") " Sep 08 02:12:46.649596 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.649571 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bb78257-f484-43ba-b84b-61365804f332" (UID: "7bb78257-f484-43ba-b84b-61365804f332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:12:46.651492 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.651470 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s" (OuterVolumeSpecName: "kube-api-access-zcm5s") pod "7bb78257-f484-43ba-b84b-61365804f332" (UID: "7bb78257-f484-43ba-b84b-61365804f332"). InnerVolumeSpecName "kube-api-access-zcm5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:12:46.750015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.749983 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca\") pod \"0c2235d0-865e-4739-939b-b4d3b57a8409\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " Sep 08 02:12:46.750015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750016 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") pod \"0c2235d0-865e-4739-939b-b4d3b57a8409\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " Sep 08 02:12:46.750250 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750036 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs\") pod \"0c2235d0-865e-4739-939b-b4d3b57a8409\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " Sep 08 02:12:46.750250 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume\") pod \"d9389835-5fbe-469b-a881-6fa52872fec6\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " Sep 08 02:12:46.750406 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750283 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98z7\" (UniqueName: \"kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7\") pod \"0c2235d0-865e-4739-939b-b4d3b57a8409\" (UID: \"0c2235d0-865e-4739-939b-b4d3b57a8409\") " Sep 08 02:12:46.750406 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnx2q\" (UniqueName: \"kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q\") pod \"d9389835-5fbe-469b-a881-6fa52872fec6\" (UID: \"d9389835-5fbe-469b-a881-6fa52872fec6\") " Sep 08 02:12:46.750508 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750398 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c2235d0-865e-4739-939b-b4d3b57a8409" (UID: "0c2235d0-865e-4739-939b-b4d3b57a8409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:12:46.750654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750630 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9389835-5fbe-469b-a881-6fa52872fec6" (UID: "d9389835-5fbe-469b-a881-6fa52872fec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:12:46.750720 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750702 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2235d0-865e-4739-939b-b4d3b57a8409-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.750720 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750716 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcm5s\" (UniqueName: \"kubernetes.io/projected/7bb78257-f484-43ba-b84b-61365804f332-kube-api-access-zcm5s\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.750834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.750730 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb78257-f484-43ba-b84b-61365804f332-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.752154 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.752129 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c2235d0-865e-4739-939b-b4d3b57a8409" (UID: "0c2235d0-865e-4739-939b-b4d3b57a8409"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:12:46.752331 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.752283 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c2235d0-865e-4739-939b-b4d3b57a8409" (UID: "0c2235d0-865e-4739-939b-b4d3b57a8409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:12:46.752396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.752308 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7" (OuterVolumeSpecName: "kube-api-access-k98z7") pod "0c2235d0-865e-4739-939b-b4d3b57a8409" (UID: "0c2235d0-865e-4739-939b-b4d3b57a8409"). InnerVolumeSpecName "kube-api-access-k98z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:12:46.752396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.752365 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q" (OuterVolumeSpecName: "kube-api-access-hnx2q") pod "d9389835-5fbe-469b-a881-6fa52872fec6" (UID: "d9389835-5fbe-469b-a881-6fa52872fec6"). InnerVolumeSpecName "kube-api-access-hnx2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:12:46.851184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.851101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98z7\" (UniqueName: \"kubernetes.io/projected/0c2235d0-865e-4739-939b-b4d3b57a8409-kube-api-access-k98z7\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.851184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.851139 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnx2q\" (UniqueName: \"kubernetes.io/projected/d9389835-5fbe-469b-a881-6fa52872fec6-kube-api-access-hnx2q\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.851184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.851149 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.851184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.851158 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2235d0-865e-4739-939b-b4d3b57a8409-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.851184 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.851167 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389835-5fbe-469b-a881-6fa52872fec6-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:12:46.977169 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.977137 2559 generic.go:356] "Generic (PLEG): container finished" podID="0c2235d0-865e-4739-939b-b4d3b57a8409" containerID="fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4" exitCode=0 Sep 08 02:12:46.977349 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.977183 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" Sep 08 02:12:46.977349 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.977222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" event={"ID":"0c2235d0-865e-4739-939b-b4d3b57a8409","Type":"ContainerDied","Data":"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4"} Sep 08 02:12:46.977349 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.977275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb" event={"ID":"0c2235d0-865e-4739-939b-b4d3b57a8409","Type":"ContainerDied","Data":"53efb8b9826885ca733a9e665aa304999d9e3df205d5f6a5c677747eaa326f4e"} Sep 08 02:12:46.977349 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.977342 2559 scope.go:117] "RemoveContainer" containerID="fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4" Sep 08 02:12:46.978601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.978579 2559 generic.go:356] "Generic (PLEG): container finished" podID="7bb78257-f484-43ba-b84b-61365804f332" containerID="5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365" exitCode=0 Sep 08 02:12:46.978721 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.978656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" event={"ID":"7bb78257-f484-43ba-b84b-61365804f332","Type":"ContainerDied","Data":"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365"} Sep 08 02:12:46.978721 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.978668 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" Sep 08 02:12:46.978721 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.978696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w" event={"ID":"7bb78257-f484-43ba-b84b-61365804f332","Type":"ContainerDied","Data":"07216b1f7b84951f95dacfe39b5e0da38717b09abf52e04edb330ed2866a3d0f"} Sep 08 02:12:46.980183 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.980162 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9389835-5fbe-469b-a881-6fa52872fec6" containerID="9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0" exitCode=0 Sep 08 02:12:46.980328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.980191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" event={"ID":"d9389835-5fbe-469b-a881-6fa52872fec6","Type":"ContainerDied","Data":"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0"} Sep 08 02:12:46.980328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.980206 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" Sep 08 02:12:46.980328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.980228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk" event={"ID":"d9389835-5fbe-469b-a881-6fa52872fec6","Type":"ContainerDied","Data":"6417a7135d8153d6608a196565e25851e9a6a932a6a74c3c6673a81a5e68cad8"} Sep 08 02:12:46.988805 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.988749 2559 scope.go:117] "RemoveContainer" containerID="fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4" Sep 08 02:12:46.989071 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:46.989047 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4\": container with ID starting with fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4 not found: ID does not exist" containerID="fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4" Sep 08 02:12:46.989149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.989082 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4"} err="failed to get container status \"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4\": rpc error: code = NotFound desc = could not find container \"fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4\": container with ID starting with fdff4896694885f5c5107d318c2a76488702df09786e420b4809cd248344c7d4 not found: ID does not exist" Sep 08 02:12:46.989149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.989105 2559 scope.go:117] "RemoveContainer" containerID="5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365" Sep 08 02:12:46.998012 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.997991 2559 scope.go:117] "RemoveContainer" containerID="5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365" Sep 08 02:12:46.998344 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:46.998325 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365\": container with ID starting with 5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365 not found: ID does not exist" containerID="5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365" Sep 08 02:12:46.998404 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.998351 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365"} err="failed to get container status \"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365\": rpc error: code = NotFound desc = could not find container \"5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365\": container with ID starting with 5cc367b496a425827d9f728064ec913679b7382b04067c4bf1b1bc149305c365 not found: ID does not exist" Sep 08 02:12:46.998404 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:46.998367 2559 scope.go:117] "RemoveContainer" containerID="9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0" Sep 08 02:12:47.006851 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.006830 2559 scope.go:117] "RemoveContainer" containerID="9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0" Sep 08 02:12:47.007115 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:12:47.007094 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0\": container with ID starting with 9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0 not found: ID does not exist" containerID="9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0" Sep 08 02:12:47.007169 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.007122 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0"} err="failed to get container status \"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0\": rpc error: code = NotFound desc = could not find container \"9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0\": container with ID starting with 9ba349eec397da834316575e1223f2b3cf5a88e514f5dc66767be4f7f36d2ba0 not found: ID does not exist" Sep 08 02:12:47.010078 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.010059 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:47.012596 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.012578 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea979531-75b74ffc54-q8znb"] Sep 08 02:12:47.025596 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.025577 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:47.029869 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.029850 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ea979531-7bdfb44c66-4nq2w"] Sep 08 02:12:47.039790 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.039772 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:47.044154 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.044134 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ea979531-5f76864bcd-kmzlk"] Sep 08 02:12:47.321016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.320978 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2235d0-865e-4739-939b-b4d3b57a8409" path="/var/lib/kubelet/pods/0c2235d0-865e-4739-939b-b4d3b57a8409/volumes" Sep 08 02:12:47.321386 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.321374 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb78257-f484-43ba-b84b-61365804f332" path="/var/lib/kubelet/pods/7bb78257-f484-43ba-b84b-61365804f332/volumes" Sep 08 02:12:47.321710 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:47.321697 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9389835-5fbe-469b-a881-6fa52872fec6" path="/var/lib/kubelet/pods/d9389835-5fbe-469b-a881-6fa52872fec6/volumes" Sep 08 02:12:50.867225 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867189 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:12:50.867854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867835 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9389835-5fbe-469b-a881-6fa52872fec6" containerName="payload-pre-processing" Sep 08 02:12:50.867958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867857 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9389835-5fbe-469b-a881-6fa52872fec6" containerName="payload-pre-processing" Sep 08 02:12:50.867958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867896 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb78257-f484-43ba-b84b-61365804f332" containerName="payload-processing" Sep 08 02:12:50.867958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867906 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb78257-f484-43ba-b84b-61365804f332" containerName="payload-processing" Sep 08 02:12:50.867958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867916 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2235d0-865e-4739-939b-b4d3b57a8409" containerName="maas-api" Sep 08 02:12:50.867958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.867925 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2235d0-865e-4739-939b-b4d3b57a8409" containerName="maas-api" Sep 08 02:12:50.868198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.868044 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9389835-5fbe-469b-a881-6fa52872fec6" containerName="payload-pre-processing" Sep 08 02:12:50.868198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.868058 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2235d0-865e-4739-939b-b4d3b57a8409" containerName="maas-api" Sep 08 02:12:50.868198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.868067 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb78257-f484-43ba-b84b-61365804f332" containerName="payload-processing" Sep 08 02:12:50.873081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.873058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:50.875175 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.875119 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c5f95cf0-serving-cert\"" Sep 08 02:12:50.879841 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.879818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:12:50.990945 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.990907 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7lbw\" (UniqueName: \"kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:50.991131 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.990969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:50.991131 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.990986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:50.991131 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:50.991077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.092469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.092428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7lbw\" (UniqueName: \"kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.092699 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.092490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.092699 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.092614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.092699 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.092662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.093267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.093237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.095082 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.095054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.095182 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.095164 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.102686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.102661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7lbw\" (UniqueName: \"kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw\") pod \"maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.186123 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.186085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:51.473588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.473561 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:12:51.475494 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:51.475465 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22783059_decf_4484_b5f1_320a587eb8d4.slice/crio-bc5aa83039eba10276b441538fea9833eb482c579469b1cc0d3d8ac0b661f0a6 WatchSource:0}: Error finding container bc5aa83039eba10276b441538fea9833eb482c579469b1cc0d3d8ac0b661f0a6: Status 404 returned error can't find the container with id bc5aa83039eba10276b441538fea9833eb482c579469b1cc0d3d8ac0b661f0a6 Sep 08 02:12:51.490617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.490593 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:12:51.495404 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.495385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.497645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.497623 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c5f95cf0\"" Sep 08 02:12:51.497739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.497663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c5f95cf0-dockercfg-n2q5k\"" Sep 08 02:12:51.502355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.502279 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:12:51.507258 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.507229 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:12:51.511432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.511414 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.529252 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.529227 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:12:51.597387 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.597356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.597533 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.597476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.597585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.597536 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxh8\" (UniqueName: \"kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.597585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.597565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxw24\" (UniqueName: \"kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.698206 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.698169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.698396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.698253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.698396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.698300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxh8\" (UniqueName: \"kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.698396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.698321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxw24\" (UniqueName: \"kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.699021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.698992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.699098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.699045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.705874 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.705856 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxh8\" (UniqueName: \"kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8\") pod \"payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:51.705959 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.705905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxw24\" (UniqueName: \"kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24\") pod \"payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.808619 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.808555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:51.821567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:51.821462 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:52.013385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.013359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" event={"ID":"22783059-decf-4484-b5f1-320a587eb8d4","Type":"ContainerStarted","Data":"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2"} Sep 08 02:12:52.013722 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.013404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" event={"ID":"22783059-decf-4484-b5f1-320a587eb8d4","Type":"ContainerStarted","Data":"bc5aa83039eba10276b441538fea9833eb482c579469b1cc0d3d8ac0b661f0a6"} Sep 08 02:12:52.013722 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.013501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:12:52.029550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.029507 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" podStartSLOduration=1.622586015 podStartE2EDuration="2.029493609s" totalImagesPullingTime="406.907594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:52.027815424 +0000 UTC m=+853.205127489" watchObservedRunningTime="2026-09-08 02:12:52.029493609 +0000 UTC m=+853.206805681" Sep 08 02:12:52.093936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.093909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:12:52.095210 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:52.095183 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6b254a_dffe_4969_b9d1_af93b2f8ed17.slice/crio-2282a7144dbdcc199e8255a831ebe67c20e666a257701f853375cb82be86ae14 WatchSource:0}: Error finding container 2282a7144dbdcc199e8255a831ebe67c20e666a257701f853375cb82be86ae14: Status 404 returned error can't find the container with id 2282a7144dbdcc199e8255a831ebe67c20e666a257701f853375cb82be86ae14 Sep 08 02:12:52.107255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:52.107226 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:12:52.110777 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:12:52.110742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c1fbac_a949_4117_b9b6_d4d123d4ca13.slice/crio-8dc0e8fbf721140dbb2271195fcda8f003d20fb161135f277818e25bc52d2e55 WatchSource:0}: Error finding container 8dc0e8fbf721140dbb2271195fcda8f003d20fb161135f277818e25bc52d2e55: Status 404 returned error can't find the container with id 8dc0e8fbf721140dbb2271195fcda8f003d20fb161135f277818e25bc52d2e55 Sep 08 02:12:53.020152 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.020094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" event={"ID":"bc6b254a-dffe-4969-b9d1-af93b2f8ed17","Type":"ContainerStarted","Data":"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6"} Sep 08 02:12:53.020603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.020278 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:12:53.020603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.020328 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" event={"ID":"bc6b254a-dffe-4969-b9d1-af93b2f8ed17","Type":"ContainerStarted","Data":"2282a7144dbdcc199e8255a831ebe67c20e666a257701f853375cb82be86ae14"} Sep 08 02:12:53.021698 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.021672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" event={"ID":"95c1fbac-a949-4117-b9b6-d4d123d4ca13","Type":"ContainerStarted","Data":"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95"} Sep 08 02:12:53.021843 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.021722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" event={"ID":"95c1fbac-a949-4117-b9b6-d4d123d4ca13","Type":"ContainerStarted","Data":"8dc0e8fbf721140dbb2271195fcda8f003d20fb161135f277818e25bc52d2e55"} Sep 08 02:12:53.021843 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.021828 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:12:53.035475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.035438 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" podStartSLOduration=2.035423944 podStartE2EDuration="2.035423944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:53.033847241 +0000 UTC m=+854.211159317" watchObservedRunningTime="2026-09-08 02:12:53.035423944 +0000 UTC m=+854.212736015" Sep 08 02:12:53.048524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:53.048478 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" podStartSLOduration=2.048461014 podStartE2EDuration="2.048461014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:53.047821519 +0000 UTC m=+854.225133592" watchObservedRunningTime="2026-09-08 02:12:53.048461014 +0000 UTC m=+854.225773088" Sep 08 02:12:58.027199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:12:58.027167 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:13:04.030906 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:04.030875 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:13:04.031363 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:04.030945 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:13:16.611780 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.611738 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:13:16.615871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.615848 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.618151 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.618128 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5d27de\"/\"kube-root-ca.crt\"" Sep 08 02:13:16.618858 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.618840 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5d27de\"/\"openshift-service-ca.crt\"" Sep 08 02:13:16.618955 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.618885 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5d27de\"/\"test-model-5d27de-kserve-self-signed-certs\"" Sep 08 02:13:16.618955 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.618844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5d27de\"/\"default-dockercfg-wl664\"" Sep 08 02:13:16.625177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.625157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:13:16.714531 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.714531 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5429w\" (UniqueName: \"kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.714531 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.714776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.714776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.714776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.714698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815147 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815339 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5429w\" (UniqueName: \"kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815339 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815339 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815339 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815339 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815657 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815705 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815669 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.815771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.815703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.817423 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.817405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.817681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.817664 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.822460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.822437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5429w\" (UniqueName: \"kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w\") pod \"test-model-5d27de-kserve-59d4f58475-b55jj\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:16.926949 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:16.926924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:17.199352 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:17.199327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:13:17.200801 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:13:17.200771 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976e5e1c_683e_49d3_9f03_110f87a8e634.slice/crio-57bdfa7201e4c370b631696189568b426c6b9ca7d54b169c96f233a20fb524c0 WatchSource:0}: Error finding container 57bdfa7201e4c370b631696189568b426c6b9ca7d54b169c96f233a20fb524c0: Status 404 returned error can't find the container with id 57bdfa7201e4c370b631696189568b426c6b9ca7d54b169c96f233a20fb524c0 Sep 08 02:13:18.182692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:18.182652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" event={"ID":"976e5e1c-683e-49d3-9f03-110f87a8e634","Type":"ContainerStarted","Data":"7774fa24cd9c7a8573456ba5cef5812e71d4d460ee9ea06394471728e85c00a0"} Sep 08 02:13:18.183074 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:18.182702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" event={"ID":"976e5e1c-683e-49d3-9f03-110f87a8e634","Type":"ContainerStarted","Data":"57bdfa7201e4c370b631696189568b426c6b9ca7d54b169c96f233a20fb524c0"} Sep 08 02:13:18.200026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:18.199969 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" podStartSLOduration=2.199952616 podStartE2EDuration="2.199952616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:18.197854577 +0000 UTC m=+879.375166648" watchObservedRunningTime="2026-09-08 02:13:18.199952616 +0000 UTC m=+879.377264688" Sep 08 02:13:21.245956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.245920 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:13:21.246695 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.246160 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" podUID="22783059-decf-4484-b5f1-320a587eb8d4" containerName="maas-api" containerID="cri-o://288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2" gracePeriod=30 Sep 08 02:13:21.299637 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.299602 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:13:21.299844 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.299817 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" podUID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" containerName="payload-processing" containerID="cri-o://d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95" gracePeriod=30 Sep 08 02:13:21.309413 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.309384 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:13:21.309670 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.309608 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" podUID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" containerName="payload-pre-processing" containerID="cri-o://bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6" gracePeriod=30 Sep 08 02:13:21.596122 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.596094 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:13:21.613682 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.613657 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:13:21.621480 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.621456 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:13:21.666180 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs\") pod \"22783059-decf-4484-b5f1-320a587eb8d4\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " Sep 08 02:13:21.666376 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls\") pod \"22783059-decf-4484-b5f1-320a587eb8d4\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " Sep 08 02:13:21.666376 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666273 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume\") pod \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " Sep 08 02:13:21.666376 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666327 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7lbw\" (UniqueName: \"kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw\") pod \"22783059-decf-4484-b5f1-320a587eb8d4\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " Sep 08 02:13:21.666376 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666355 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume\") pod \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " Sep 08 02:13:21.666609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqxh8\" (UniqueName: \"kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8\") pod \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\" (UID: \"95c1fbac-a949-4117-b9b6-d4d123d4ca13\") " Sep 08 02:13:21.666609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666407 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca\") pod \"22783059-decf-4484-b5f1-320a587eb8d4\" (UID: \"22783059-decf-4484-b5f1-320a587eb8d4\") " Sep 08 02:13:21.666609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666457 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxw24\" (UniqueName: \"kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24\") pod \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\" (UID: \"bc6b254a-dffe-4969-b9d1-af93b2f8ed17\") " Sep 08 02:13:21.666983 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.666946 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95c1fbac-a949-4117-b9b6-d4d123d4ca13" (UID: "95c1fbac-a949-4117-b9b6-d4d123d4ca13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:21.667310 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.667251 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22783059-decf-4484-b5f1-320a587eb8d4" (UID: "22783059-decf-4484-b5f1-320a587eb8d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:21.667934 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.667904 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc6b254a-dffe-4969-b9d1-af93b2f8ed17" (UID: "bc6b254a-dffe-4969-b9d1-af93b2f8ed17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:21.669238 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.669207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22783059-decf-4484-b5f1-320a587eb8d4" (UID: "22783059-decf-4484-b5f1-320a587eb8d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:21.669238 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.669222 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24" (OuterVolumeSpecName: "kube-api-access-mxw24") pod "bc6b254a-dffe-4969-b9d1-af93b2f8ed17" (UID: "bc6b254a-dffe-4969-b9d1-af93b2f8ed17"). InnerVolumeSpecName "kube-api-access-mxw24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:21.669385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.669345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw" (OuterVolumeSpecName: "kube-api-access-g7lbw") pod "22783059-decf-4484-b5f1-320a587eb8d4" (UID: "22783059-decf-4484-b5f1-320a587eb8d4"). InnerVolumeSpecName "kube-api-access-g7lbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:21.669869 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.669844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8" (OuterVolumeSpecName: "kube-api-access-xqxh8") pod "95c1fbac-a949-4117-b9b6-d4d123d4ca13" (UID: "95c1fbac-a949-4117-b9b6-d4d123d4ca13"). InnerVolumeSpecName "kube-api-access-xqxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:21.670533 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.670515 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22783059-decf-4484-b5f1-320a587eb8d4" (UID: "22783059-decf-4484-b5f1-320a587eb8d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.767977 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768005 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22783059-decf-4484-b5f1-320a587eb8d4-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768015 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768025 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7lbw\" (UniqueName: \"kubernetes.io/projected/22783059-decf-4484-b5f1-320a587eb8d4-kube-api-access-g7lbw\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768036 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c1fbac-a949-4117-b9b6-d4d123d4ca13-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768044 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqxh8\" (UniqueName: \"kubernetes.io/projected/95c1fbac-a949-4117-b9b6-d4d123d4ca13-kube-api-access-xqxh8\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768052 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22783059-decf-4484-b5f1-320a587eb8d4-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:21.768398 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:21.768062 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxw24\" (UniqueName: \"kubernetes.io/projected/bc6b254a-dffe-4969-b9d1-af93b2f8ed17-kube-api-access-mxw24\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:22.210759 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.210720 2559 generic.go:356] "Generic (PLEG): container finished" podID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" containerID="bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6" exitCode=0 Sep 08 02:13:22.210950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.210764 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" Sep 08 02:13:22.210950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.210803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" event={"ID":"bc6b254a-dffe-4969-b9d1-af93b2f8ed17","Type":"ContainerDied","Data":"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6"} Sep 08 02:13:22.210950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.210851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx" event={"ID":"bc6b254a-dffe-4969-b9d1-af93b2f8ed17","Type":"ContainerDied","Data":"2282a7144dbdcc199e8255a831ebe67c20e666a257701f853375cb82be86ae14"} Sep 08 02:13:22.210950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.210875 2559 scope.go:117] "RemoveContainer" containerID="bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6" Sep 08 02:13:22.212424 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.212399 2559 generic.go:356] "Generic (PLEG): container finished" podID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" containerID="d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95" exitCode=0 Sep 08 02:13:22.212537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.212450 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" Sep 08 02:13:22.212537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.212480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" event={"ID":"95c1fbac-a949-4117-b9b6-d4d123d4ca13","Type":"ContainerDied","Data":"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95"} Sep 08 02:13:22.212645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.212537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh" event={"ID":"95c1fbac-a949-4117-b9b6-d4d123d4ca13","Type":"ContainerDied","Data":"8dc0e8fbf721140dbb2271195fcda8f003d20fb161135f277818e25bc52d2e55"} Sep 08 02:13:22.214158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.214132 2559 generic.go:356] "Generic (PLEG): container finished" podID="22783059-decf-4484-b5f1-320a587eb8d4" containerID="288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2" exitCode=0 Sep 08 02:13:22.214158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.214166 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" Sep 08 02:13:22.214370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.214222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" event={"ID":"22783059-decf-4484-b5f1-320a587eb8d4","Type":"ContainerDied","Data":"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2"} Sep 08 02:13:22.214370 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.214260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j" event={"ID":"22783059-decf-4484-b5f1-320a587eb8d4","Type":"ContainerDied","Data":"bc5aa83039eba10276b441538fea9833eb482c579469b1cc0d3d8ac0b661f0a6"} Sep 08 02:13:22.221958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.221937 2559 scope.go:117] "RemoveContainer" containerID="bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6" Sep 08 02:13:22.222402 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:22.222252 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6\": container with ID starting with bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6 not found: ID does not exist" containerID="bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6" Sep 08 02:13:22.222402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.222309 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6"} err="failed to get container status \"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6\": rpc error: code = NotFound desc = could not find container \"bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6\": container with ID starting with bfed03a3cb92937687ddfd63a5982221c14074a9ed81fc78a2360826159ea6a6 not found: ID does not exist" Sep 08 02:13:22.222402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.222334 2559 scope.go:117] "RemoveContainer" containerID="d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95" Sep 08 02:13:22.231143 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.231128 2559 scope.go:117] "RemoveContainer" containerID="d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95" Sep 08 02:13:22.231448 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:22.231430 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95\": container with ID starting with d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95 not found: ID does not exist" containerID="d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95" Sep 08 02:13:22.231530 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.231452 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95"} err="failed to get container status \"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95\": rpc error: code = NotFound desc = could not find container \"d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95\": container with ID starting with d064c43896a7f64230727a23a378f3620c2ff219f62fe9eeaefd3bf6d97d8e95 not found: ID does not exist" Sep 08 02:13:22.231530 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.231467 2559 scope.go:117] "RemoveContainer" containerID="288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2" Sep 08 02:13:22.237967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.237938 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:13:22.244072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.244049 2559 scope.go:117] "RemoveContainer" containerID="288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2" Sep 08 02:13:22.244606 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:22.244584 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2\": container with ID starting with 288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2 not found: ID does not exist" containerID="288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2" Sep 08 02:13:22.244700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.244612 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2"} err="failed to get container status \"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2\": rpc error: code = NotFound desc = could not find container \"288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2\": container with ID starting with 288a35df76f301edbf4b3af3d1bfcada1200b95930671b1877e1703a062053b2 not found: ID does not exist" Sep 08 02:13:22.245103 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.245086 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5f95cf0-6895b84558-plxdx"] Sep 08 02:13:22.256622 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.256599 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:13:22.261388 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.261364 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5f95cf0-8549b78cb-9wm8j"] Sep 08 02:13:22.271766 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.271743 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:13:22.274896 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:22.274873 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5f95cf0-6cf9bb878f-r9hhh"] Sep 08 02:13:23.322214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:23.322171 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22783059-decf-4484-b5f1-320a587eb8d4" path="/var/lib/kubelet/pods/22783059-decf-4484-b5f1-320a587eb8d4/volumes" Sep 08 02:13:23.322708 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:23.322659 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" path="/var/lib/kubelet/pods/95c1fbac-a949-4117-b9b6-d4d123d4ca13/volumes" Sep 08 02:13:23.323026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:23.323010 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" path="/var/lib/kubelet/pods/bc6b254a-dffe-4969-b9d1-af93b2f8ed17/volumes" Sep 08 02:13:26.927770 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:26.927736 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:26.928217 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:26.927781 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:26.941229 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:26.941199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:27.089324 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089277 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:27.089736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089719 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22783059-decf-4484-b5f1-320a587eb8d4" containerName="maas-api" Sep 08 02:13:27.089736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089735 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22783059-decf-4484-b5f1-320a587eb8d4" containerName="maas-api" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089746 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" containerName="payload-processing" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089752 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" containerName="payload-processing" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089760 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" containerName="payload-pre-processing" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089767 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" containerName="payload-pre-processing" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089828 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22783059-decf-4484-b5f1-320a587eb8d4" containerName="maas-api" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089835 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6b254a-dffe-4969-b9d1-af93b2f8ed17" containerName="payload-pre-processing" Sep 08 02:13:27.089915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.089843 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c1fbac-a949-4117-b9b6-d4d123d4ca13" containerName="payload-processing" Sep 08 02:13:27.094142 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.094122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.096253 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.096233 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b1ffefe6-serving-cert\"" Sep 08 02:13:27.102172 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.102150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:27.220679 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.220616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.220679 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.220653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.220824 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.220697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz84\" (UniqueName: \"kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.220864 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.220827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.269573 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.269542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:13:27.321403 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.321369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.321603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.321431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.321603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.321455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.321603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.321511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz84\" (UniqueName: \"kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.322253 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.322224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.323830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.323809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.324226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.324207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.328878 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.328852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz84\" (UniqueName: \"kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84\") pod \"maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.427062 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.427031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:27.698053 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.698015 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:27.703103 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.703079 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.705384 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.705367 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b1ffefe6-dockercfg-pls7x\"" Sep 08 02:13:27.705460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.705372 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b1ffefe6\"" Sep 08 02:13:27.711483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.711460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:27.715328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.715301 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:27.715867 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:13:27.715842 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069cb294_207d_47d7_bdc2_6d931925e67b.slice/crio-506d6ecb509bbc09f0f06e47cabacc4660d88b217aff76fd80d90deda9b27e9e WatchSource:0}: Error finding container 506d6ecb509bbc09f0f06e47cabacc4660d88b217aff76fd80d90deda9b27e9e: Status 404 returned error can't find the container with id 506d6ecb509bbc09f0f06e47cabacc4660d88b217aff76fd80d90deda9b27e9e Sep 08 02:13:27.737162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.737072 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:27.742634 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.742599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.745834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.745804 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:27.827154 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.827112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtj5x\" (UniqueName: \"kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.827365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.827165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.827365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.827223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dq9k\" (UniqueName: \"kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.827455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.827398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.928610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.928574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtj5x\" (UniqueName: \"kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.928610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.928613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.929068 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.928629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dq9k\" (UniqueName: \"kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.929068 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.928716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.929569 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.929545 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.929619 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.929570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:27.935986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.935960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtj5x\" (UniqueName: \"kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x\") pod \"payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:27.935986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:27.935983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dq9k\" (UniqueName: \"kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k\") pod \"payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:28.017806 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.017730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:28.054156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.054123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:28.266071 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.266038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" event={"ID":"069cb294-207d-47d7-bdc2-6d931925e67b","Type":"ContainerStarted","Data":"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172"} Sep 08 02:13:28.266227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.266100 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" event={"ID":"069cb294-207d-47d7-bdc2-6d931925e67b","Type":"ContainerStarted","Data":"506d6ecb509bbc09f0f06e47cabacc4660d88b217aff76fd80d90deda9b27e9e"} Sep 08 02:13:28.266332 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.266281 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:28.283251 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.283175 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" podStartSLOduration=0.81753044 podStartE2EDuration="1.283160449s" totalImagesPullingTime="465.630009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:28.281346317 +0000 UTC m=+889.458658381" watchObservedRunningTime="2026-09-08 02:13:28.283160449 +0000 UTC m=+889.460472520" Sep 08 02:13:28.295010 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.294988 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:28.297274 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:13:28.297253 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7785b2_38a8_4e02_a273_7433d4faec0f.slice/crio-ed8be5be02e7694414cb68aa760d450e9538407bdb3f16d84ccd4dd9d8af8c13 WatchSource:0}: Error finding container ed8be5be02e7694414cb68aa760d450e9538407bdb3f16d84ccd4dd9d8af8c13: Status 404 returned error can't find the container with id ed8be5be02e7694414cb68aa760d450e9538407bdb3f16d84ccd4dd9d8af8c13 Sep 08 02:13:28.336201 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:28.336173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:28.337073 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:13:28.337052 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441cafd0_44ef_4584_a8e8_2c5defb4782c.slice/crio-3810f94bbe9afea3c286d27d63fc605a4fb45b98276d972d93b49eb690071664 WatchSource:0}: Error finding container 3810f94bbe9afea3c286d27d63fc605a4fb45b98276d972d93b49eb690071664: Status 404 returned error can't find the container with id 3810f94bbe9afea3c286d27d63fc605a4fb45b98276d972d93b49eb690071664 Sep 08 02:13:29.273928 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.273892 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" event={"ID":"441cafd0-44ef-4584-a8e8-2c5defb4782c","Type":"ContainerStarted","Data":"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7"} Sep 08 02:13:29.274414 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.273949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" event={"ID":"441cafd0-44ef-4584-a8e8-2c5defb4782c","Type":"ContainerStarted","Data":"3810f94bbe9afea3c286d27d63fc605a4fb45b98276d972d93b49eb690071664"} Sep 08 02:13:29.274414 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.273970 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:29.275967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.275933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" event={"ID":"5b7785b2-38a8-4e02-a273-7433d4faec0f","Type":"ContainerStarted","Data":"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159"} Sep 08 02:13:29.276112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.275979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" event={"ID":"5b7785b2-38a8-4e02-a273-7433d4faec0f","Type":"ContainerStarted","Data":"ed8be5be02e7694414cb68aa760d450e9538407bdb3f16d84ccd4dd9d8af8c13"} Sep 08 02:13:29.276112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.276019 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:29.290241 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.290191 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" podStartSLOduration=2.290174015 podStartE2EDuration="2.290174015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:29.287992205 +0000 UTC m=+890.465304278" watchObservedRunningTime="2026-09-08 02:13:29.290174015 +0000 UTC m=+890.467486089" Sep 08 02:13:29.303879 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:29.303835 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" podStartSLOduration=2.303821483 podStartE2EDuration="2.303821483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:29.301820183 +0000 UTC m=+890.479132256" watchObservedRunningTime="2026-09-08 02:13:29.303821483 +0000 UTC m=+890.481133554" Sep 08 02:13:30.667432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:30.667397 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:13:30.667863 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:30.667630 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" podUID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" containerName="maas-api" containerID="cri-o://96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3" gracePeriod=30 Sep 08 02:13:30.932853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:30.932790 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:13:31.063884 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.063841 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") pod \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " Sep 08 02:13:31.064086 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.063892 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca\") pod \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " Sep 08 02:13:31.064086 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.064047 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n\") pod \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " Sep 08 02:13:31.064086 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.064074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs\") pod \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\" (UID: \"b648f20c-a699-41d1-bf4e-e9cb7f35c04f\") " Sep 08 02:13:31.064381 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.064353 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b648f20c-a699-41d1-bf4e-e9cb7f35c04f" (UID: "b648f20c-a699-41d1-bf4e-e9cb7f35c04f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:31.064465 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.064453 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:31.066247 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.066215 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b648f20c-a699-41d1-bf4e-e9cb7f35c04f" (UID: "b648f20c-a699-41d1-bf4e-e9cb7f35c04f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:31.066563 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.066542 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n" (OuterVolumeSpecName: "kube-api-access-cwh2n") pod "b648f20c-a699-41d1-bf4e-e9cb7f35c04f" (UID: "b648f20c-a699-41d1-bf4e-e9cb7f35c04f"). InnerVolumeSpecName "kube-api-access-cwh2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:31.066630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.066552 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b648f20c-a699-41d1-bf4e-e9cb7f35c04f" (UID: "b648f20c-a699-41d1-bf4e-e9cb7f35c04f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:31.165586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.165550 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-kube-api-access-cwh2n\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:31.165586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.165581 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:31.165586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.165591 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b648f20c-a699-41d1-bf4e-e9cb7f35c04f-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:31.290672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.290581 2559 generic.go:356] "Generic (PLEG): container finished" podID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" containerID="96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3" exitCode=0 Sep 08 02:13:31.290672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.290631 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" event={"ID":"b648f20c-a699-41d1-bf4e-e9cb7f35c04f","Type":"ContainerDied","Data":"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3"} Sep 08 02:13:31.290672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.290639 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" Sep 08 02:13:31.290901 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.290674 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb" event={"ID":"b648f20c-a699-41d1-bf4e-e9cb7f35c04f","Type":"ContainerDied","Data":"0cdf6ab5b143aa5384bdd9ed6eef875dbbd5710697e126d900284ae3af3ab314"} Sep 08 02:13:31.290901 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.290696 2559 scope.go:117] "RemoveContainer" containerID="96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3" Sep 08 02:13:31.301364 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.301346 2559 scope.go:117] "RemoveContainer" containerID="96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3" Sep 08 02:13:31.301689 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:31.301672 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3\": container with ID starting with 96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3 not found: ID does not exist" containerID="96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3" Sep 08 02:13:31.301812 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.301698 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3"} err="failed to get container status \"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3\": rpc error: code = NotFound desc = could not find container \"96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3\": container with ID starting with 96ef512d59e259ceec7446a71a9aa4adac6e08ed1d6affd7106e6e2df394fdd3 not found: ID does not exist" Sep 08 02:13:31.320750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.320717 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:13:31.320750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:31.320753 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-308137-556f6d8f8b-z6bjb"] Sep 08 02:13:33.319433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:33.319400 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" path="/var/lib/kubelet/pods/b648f20c-a699-41d1-bf4e-e9cb7f35c04f/volumes" Sep 08 02:13:34.281551 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:34.281523 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:39.347329 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:39.347281 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:13:39.347782 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:39.347283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:13:40.285284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:40.285255 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:40.285518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:40.285498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:41.902513 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.902475 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:13:41.903267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.903236 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" containerName="maas-api" Sep 08 02:13:41.903267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.903266 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" containerName="maas-api" Sep 08 02:13:41.903445 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.903408 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b648f20c-a699-41d1-bf4e-e9cb7f35c04f" containerName="maas-api" Sep 08 02:13:41.916278 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.916243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:41.919444 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.919417 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:13:41.921915 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:41.921897 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-173970-serving-cert\"" Sep 08 02:13:42.075353 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.075317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.075546 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.075371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.075546 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.075450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgln7\" (UniqueName: \"kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.075546 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.075508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.177018 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.176947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgln7\" (UniqueName: \"kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.177018 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.177004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.177248 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.177074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.177248 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.177092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.177748 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.177723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.179783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.179757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.180128 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.180102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.184976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.184946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgln7\" (UniqueName: \"kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7\") pod \"maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.228880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.228846 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:42.504624 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:42.504594 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:13:42.506021 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:13:42.506000 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd0dab9_6a90_4192_ac39_a051aa7f93cc.slice/crio-2cb52f316436f3e721d78f692d7d05fc01a3f87674ecc3ed1b6944973bc1b91a WatchSource:0}: Error finding container 2cb52f316436f3e721d78f692d7d05fc01a3f87674ecc3ed1b6944973bc1b91a: Status 404 returned error can't find the container with id 2cb52f316436f3e721d78f692d7d05fc01a3f87674ecc3ed1b6944973bc1b91a Sep 08 02:13:43.371428 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:43.371342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" event={"ID":"6bd0dab9-6a90-4192-ac39-a051aa7f93cc","Type":"ContainerStarted","Data":"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a"} Sep 08 02:13:43.371428 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:43.371405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" event={"ID":"6bd0dab9-6a90-4192-ac39-a051aa7f93cc","Type":"ContainerStarted","Data":"2cb52f316436f3e721d78f692d7d05fc01a3f87674ecc3ed1b6944973bc1b91a"} Sep 08 02:13:43.371861 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:43.371553 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:43.390020 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:43.389973 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" podStartSLOduration=1.990636788 podStartE2EDuration="2.389957806s" totalImagesPullingTime="399.321018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:43.387359193 +0000 UTC m=+904.564671262" watchObservedRunningTime="2026-09-08 02:13:43.389957806 +0000 UTC m=+904.567269878" Sep 08 02:13:49.382816 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:49.382785 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:13:57.684937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.684897 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:57.685396 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.685163 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" podUID="069cb294-207d-47d7-bdc2-6d931925e67b" containerName="maas-api" containerID="cri-o://b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172" gracePeriod=30 Sep 08 02:13:57.741361 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.741328 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:57.741566 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.741537 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" podUID="441cafd0-44ef-4584-a8e8-2c5defb4782c" containerName="payload-processing" containerID="cri-o://621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7" gracePeriod=30 Sep 08 02:13:57.747505 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.747425 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:57.747893 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:57.747728 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" podUID="5b7785b2-38a8-4e02-a273-7433d4faec0f" containerName="payload-pre-processing" containerID="cri-o://72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159" gracePeriod=30 Sep 08 02:13:58.044308 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.044270 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:58.091953 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.091922 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:58.096279 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.096252 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.123838 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fz84\" (UniqueName: \"kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84\") pod \"069cb294-207d-47d7-bdc2-6d931925e67b\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.123896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs\") pod \"069cb294-207d-47d7-bdc2-6d931925e67b\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.123955 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume\") pod \"5b7785b2-38a8-4e02-a273-7433d4faec0f\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.123982 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume\") pod \"441cafd0-44ef-4584-a8e8-2c5defb4782c\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124022 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtj5x\" (UniqueName: \"kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x\") pod \"441cafd0-44ef-4584-a8e8-2c5defb4782c\" (UID: \"441cafd0-44ef-4584-a8e8-2c5defb4782c\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124067 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls\") pod \"069cb294-207d-47d7-bdc2-6d931925e67b\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124113 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca\") pod \"069cb294-207d-47d7-bdc2-6d931925e67b\" (UID: \"069cb294-207d-47d7-bdc2-6d931925e67b\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124175 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dq9k\" (UniqueName: \"kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k\") pod \"5b7785b2-38a8-4e02-a273-7433d4faec0f\" (UID: \"5b7785b2-38a8-4e02-a273-7433d4faec0f\") " Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b7785b2-38a8-4e02-a273-7433d4faec0f" (UID: "5b7785b2-38a8-4e02-a273-7433d4faec0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:58.124714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.124580 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7785b2-38a8-4e02-a273-7433d4faec0f-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.126254 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.125703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "069cb294-207d-47d7-bdc2-6d931925e67b" (UID: "069cb294-207d-47d7-bdc2-6d931925e67b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:58.126254 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.126173 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "441cafd0-44ef-4584-a8e8-2c5defb4782c" (UID: "441cafd0-44ef-4584-a8e8-2c5defb4782c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:13:58.127365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.127309 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x" (OuterVolumeSpecName: "kube-api-access-xtj5x") pod "441cafd0-44ef-4584-a8e8-2c5defb4782c" (UID: "441cafd0-44ef-4584-a8e8-2c5defb4782c"). InnerVolumeSpecName "kube-api-access-xtj5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:58.127569 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.127546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84" (OuterVolumeSpecName: "kube-api-access-6fz84") pod "069cb294-207d-47d7-bdc2-6d931925e67b" (UID: "069cb294-207d-47d7-bdc2-6d931925e67b"). InnerVolumeSpecName "kube-api-access-6fz84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:58.127983 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.127949 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "069cb294-207d-47d7-bdc2-6d931925e67b" (UID: "069cb294-207d-47d7-bdc2-6d931925e67b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:58.128690 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.128669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k" (OuterVolumeSpecName: "kube-api-access-7dq9k") pod "5b7785b2-38a8-4e02-a273-7433d4faec0f" (UID: "5b7785b2-38a8-4e02-a273-7433d4faec0f"). InnerVolumeSpecName "kube-api-access-7dq9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:13:58.129114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.129093 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "069cb294-207d-47d7-bdc2-6d931925e67b" (UID: "069cb294-207d-47d7-bdc2-6d931925e67b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225855 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz84\" (UniqueName: \"kubernetes.io/projected/069cb294-207d-47d7-bdc2-6d931925e67b-kube-api-access-6fz84\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225886 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225896 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441cafd0-44ef-4584-a8e8-2c5defb4782c-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225905 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtj5x\" (UniqueName: \"kubernetes.io/projected/441cafd0-44ef-4584-a8e8-2c5defb4782c-kube-api-access-xtj5x\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225915 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069cb294-207d-47d7-bdc2-6d931925e67b-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225923 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069cb294-207d-47d7-bdc2-6d931925e67b-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.225942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.225934 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dq9k\" (UniqueName: \"kubernetes.io/projected/5b7785b2-38a8-4e02-a273-7433d4faec0f-kube-api-access-7dq9k\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:13:58.463401 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.463360 2559 generic.go:356] "Generic (PLEG): container finished" podID="069cb294-207d-47d7-bdc2-6d931925e67b" containerID="b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172" exitCode=0 Sep 08 02:13:58.463594 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.463415 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" Sep 08 02:13:58.463594 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.463447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" event={"ID":"069cb294-207d-47d7-bdc2-6d931925e67b","Type":"ContainerDied","Data":"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172"} Sep 08 02:13:58.463594 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.463514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq" event={"ID":"069cb294-207d-47d7-bdc2-6d931925e67b","Type":"ContainerDied","Data":"506d6ecb509bbc09f0f06e47cabacc4660d88b217aff76fd80d90deda9b27e9e"} Sep 08 02:13:58.463594 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.463542 2559 scope.go:117] "RemoveContainer" containerID="b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172" Sep 08 02:13:58.465161 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.465138 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b7785b2-38a8-4e02-a273-7433d4faec0f" containerID="72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159" exitCode=0 Sep 08 02:13:58.465265 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.465181 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" Sep 08 02:13:58.465265 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.465227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" event={"ID":"5b7785b2-38a8-4e02-a273-7433d4faec0f","Type":"ContainerDied","Data":"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159"} Sep 08 02:13:58.465408 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.465304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9" event={"ID":"5b7785b2-38a8-4e02-a273-7433d4faec0f","Type":"ContainerDied","Data":"ed8be5be02e7694414cb68aa760d450e9538407bdb3f16d84ccd4dd9d8af8c13"} Sep 08 02:13:58.466930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.466907 2559 generic.go:356] "Generic (PLEG): container finished" podID="441cafd0-44ef-4584-a8e8-2c5defb4782c" containerID="621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7" exitCode=0 Sep 08 02:13:58.467042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.466940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" event={"ID":"441cafd0-44ef-4584-a8e8-2c5defb4782c","Type":"ContainerDied","Data":"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7"} Sep 08 02:13:58.467117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.467042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" event={"ID":"441cafd0-44ef-4584-a8e8-2c5defb4782c","Type":"ContainerDied","Data":"3810f94bbe9afea3c286d27d63fc605a4fb45b98276d972d93b49eb690071664"} Sep 08 02:13:58.467117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.466957 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9" Sep 08 02:13:58.479347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.475983 2559 scope.go:117] "RemoveContainer" containerID="b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172" Sep 08 02:13:58.479347 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:58.476702 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172\": container with ID starting with b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172 not found: ID does not exist" containerID="b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172" Sep 08 02:13:58.479347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.476740 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172"} err="failed to get container status \"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172\": rpc error: code = NotFound desc = could not find container \"b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172\": container with ID starting with b60ad0e575bf57fddb5f7f56f0cbfd664afc174830dd8e85705eaf4ea2c5a172 not found: ID does not exist" Sep 08 02:13:58.479347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.476771 2559 scope.go:117] "RemoveContainer" containerID="72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159" Sep 08 02:13:58.488623 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.488601 2559 scope.go:117] "RemoveContainer" containerID="72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159" Sep 08 02:13:58.488922 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:58.488905 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159\": container with ID starting with 72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159 not found: ID does not exist" containerID="72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159" Sep 08 02:13:58.488969 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.488930 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159"} err="failed to get container status \"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159\": rpc error: code = NotFound desc = could not find container \"72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159\": container with ID starting with 72b8714040add984681c5c71eae348046203704207c85da22c27e7fec213c159 not found: ID does not exist" Sep 08 02:13:58.488969 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.488946 2559 scope.go:117] "RemoveContainer" containerID="621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7" Sep 08 02:13:58.498667 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.498635 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:58.499171 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.499154 2559 scope.go:117] "RemoveContainer" containerID="621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7" Sep 08 02:13:58.499508 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:13:58.499490 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7\": container with ID starting with 621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7 not found: ID does not exist" containerID="621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7" Sep 08 02:13:58.499580 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.499514 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7"} err="failed to get container status \"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7\": rpc error: code = NotFound desc = could not find container \"621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7\": container with ID starting with 621b3dd866d5b515c9f2ef813f057a0f026c3f8ebcdc72583088b1eca8200af7 not found: ID does not exist" Sep 08 02:13:58.501484 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.501461 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b1ffefe6-5684f97f8b-2txz9"] Sep 08 02:13:58.507982 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.507944 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:58.510409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.510388 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b1ffefe6-67478f8769-xkmh9"] Sep 08 02:13:58.522729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.522696 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:58.525974 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:58.525949 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ffefe6-75cb4ff67-ktjgq"] Sep 08 02:13:59.320727 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:59.320692 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="069cb294-207d-47d7-bdc2-6d931925e67b" path="/var/lib/kubelet/pods/069cb294-207d-47d7-bdc2-6d931925e67b/volumes" Sep 08 02:13:59.321094 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:59.321048 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441cafd0-44ef-4584-a8e8-2c5defb4782c" path="/var/lib/kubelet/pods/441cafd0-44ef-4584-a8e8-2c5defb4782c/volumes" Sep 08 02:13:59.321379 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:13:59.321365 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7785b2-38a8-4e02-a273-7433d4faec0f" path="/var/lib/kubelet/pods/5b7785b2-38a8-4e02-a273-7433d4faec0f/volumes" Sep 08 02:15:12.092890 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.092799 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:15:12.093504 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.093058 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" podUID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" containerName="maas-api" containerID="cri-o://ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a" gracePeriod=30 Sep 08 02:15:12.363338 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.363312 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:15:12.509938 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.509907 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgln7\" (UniqueName: \"kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7\") pod \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " Sep 08 02:15:12.510107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.509947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls\") pod \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " Sep 08 02:15:12.510107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.509979 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca\") pod \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " Sep 08 02:15:12.510107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.510042 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs\") pod \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\" (UID: \"6bd0dab9-6a90-4192-ac39-a051aa7f93cc\") " Sep 08 02:15:12.510385 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.510357 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bd0dab9-6a90-4192-ac39-a051aa7f93cc" (UID: "6bd0dab9-6a90-4192-ac39-a051aa7f93cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:15:12.512202 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.512168 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6bd0dab9-6a90-4192-ac39-a051aa7f93cc" (UID: "6bd0dab9-6a90-4192-ac39-a051aa7f93cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:15:12.512313 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.512210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7" (OuterVolumeSpecName: "kube-api-access-fgln7") pod "6bd0dab9-6a90-4192-ac39-a051aa7f93cc" (UID: "6bd0dab9-6a90-4192-ac39-a051aa7f93cc"). InnerVolumeSpecName "kube-api-access-fgln7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:15:12.512313 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.512206 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bd0dab9-6a90-4192-ac39-a051aa7f93cc" (UID: "6bd0dab9-6a90-4192-ac39-a051aa7f93cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:15:12.611732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.611647 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgln7\" (UniqueName: \"kubernetes.io/projected/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-kube-api-access-fgln7\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:12.611732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.611679 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:12.611732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.611690 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:12.611732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.611701 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd0dab9-6a90-4192-ac39-a051aa7f93cc-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:12.914542 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.914511 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" containerID="ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a" exitCode=0 Sep 08 02:15:12.914718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.914557 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" Sep 08 02:15:12.914718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.914587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" event={"ID":"6bd0dab9-6a90-4192-ac39-a051aa7f93cc","Type":"ContainerDied","Data":"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a"} Sep 08 02:15:12.914718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.914640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk" event={"ID":"6bd0dab9-6a90-4192-ac39-a051aa7f93cc","Type":"ContainerDied","Data":"2cb52f316436f3e721d78f692d7d05fc01a3f87674ecc3ed1b6944973bc1b91a"} Sep 08 02:15:12.914718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.914671 2559 scope.go:117] "RemoveContainer" containerID="ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a" Sep 08 02:15:12.924686 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.924668 2559 scope.go:117] "RemoveContainer" containerID="ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a" Sep 08 02:15:12.924960 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:15:12.924944 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a\": container with ID starting with ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a not found: ID does not exist" containerID="ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a" Sep 08 02:15:12.925014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.924967 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a"} err="failed to get container status \"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a\": rpc error: code = NotFound desc = could not find container \"ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a\": container with ID starting with ca6376b3f4176a5eda199a206ef91b4c00f61143e35a22f0231e17745209084a not found: ID does not exist" Sep 08 02:15:12.937103 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.937075 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:15:12.941395 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:12.941373 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-173970-7c4ff4ccc8-j5gkk"] Sep 08 02:15:13.319962 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:13.319883 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" path="/var/lib/kubelet/pods/6bd0dab9-6a90-4192-ac39-a051aa7f93cc/volumes" Sep 08 02:15:19.747560 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747529 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:19.747994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747953 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441cafd0-44ef-4584-a8e8-2c5defb4782c" containerName="payload-processing" Sep 08 02:15:19.747994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747966 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="441cafd0-44ef-4584-a8e8-2c5defb4782c" containerName="payload-processing" Sep 08 02:15:19.747994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747979 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7785b2-38a8-4e02-a273-7433d4faec0f" containerName="payload-pre-processing" Sep 08 02:15:19.747994 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747987 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7785b2-38a8-4e02-a273-7433d4faec0f" containerName="payload-pre-processing" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.747997 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748005 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748014 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069cb294-207d-47d7-bdc2-6d931925e67b" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748019 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="069cb294-207d-47d7-bdc2-6d931925e67b" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748089 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="069cb294-207d-47d7-bdc2-6d931925e67b" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748097 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7785b2-38a8-4e02-a273-7433d4faec0f" containerName="payload-pre-processing" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748103 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd0dab9-6a90-4192-ac39-a051aa7f93cc" containerName="maas-api" Sep 08 02:15:19.748135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.748111 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="441cafd0-44ef-4584-a8e8-2c5defb4782c" containerName="payload-processing" Sep 08 02:15:19.752754 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.752736 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.755057 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.755033 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-cf06cb-openshift-default-dockercfg-gxcbs\"" Sep 08 02:15:19.755176 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.755088 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 02:15:19.764269 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.764245 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:19.874240 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgcb\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.874666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.874628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975372 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975563 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975563 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgcb\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.975854 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.975805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.976098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.976026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.976197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.976161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.976459 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.976441 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.977888 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.977864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.978284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.978265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.982368 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.982340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:19.982843 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:19.982825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgcb\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb\") pod \"e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:20.065428 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:20.065343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:20.373620 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:15:20.373589 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5379ee74_b22b_4bc1_9700_cd0caec621d4.slice/crio-8e39d0deae961ed73d9c7f3185ca842f407f041a2659f3ccaaf8712d08ec93fc WatchSource:0}: Error finding container 8e39d0deae961ed73d9c7f3185ca842f407f041a2659f3ccaaf8712d08ec93fc: Status 404 returned error can't find the container with id 8e39d0deae961ed73d9c7f3185ca842f407f041a2659f3ccaaf8712d08ec93fc Sep 08 02:15:20.374162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:20.374139 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:20.963659 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:20.963622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" event={"ID":"5379ee74-b22b-4bc1-9700-cd0caec621d4","Type":"ContainerStarted","Data":"8e39d0deae961ed73d9c7f3185ca842f407f041a2659f3ccaaf8712d08ec93fc"} Sep 08 02:15:23.433564 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:23.433530 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:15:23.433840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:23.433601 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:15:23.433840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:23.433629 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:15:23.986105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:23.986074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" event={"ID":"5379ee74-b22b-4bc1-9700-cd0caec621d4","Type":"ContainerStarted","Data":"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b"} Sep 08 02:15:24.006669 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:24.006621 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" podStartSLOduration=1.948986622 podStartE2EDuration="5.006605557s" totalImagesPullingTime="3.057618935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:15:24.003897566 +0000 UTC m=+1005.181209652" watchObservedRunningTime="2026-09-08 02:15:24.006605557 +0000 UTC m=+1005.183917628" Sep 08 02:15:24.066363 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:24.066333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:24.071407 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:24.071384 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:24.992302 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:24.992262 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:24.993226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:24.993205 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:36.507487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.507452 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:15:36.514791 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.514770 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.516976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.516954 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8fa74948-serving-cert\"" Sep 08 02:15:36.524554 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.524528 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:15:36.618515 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.618479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ps6\" (UniqueName: \"kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.618515 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.618518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.618724 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.618583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.618724 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.618621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.719593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.719554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.719789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.719601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.719789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.719705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ps6\" (UniqueName: \"kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.719789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.719740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.719966 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:15:36.719857 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-serving-cert: secret "maas-api-e2e-mt-8fa74948-serving-cert" not found Sep 08 02:15:36.719966 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:15:36.719939 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls podName:a5597d72-86a8-4144-b8c7-dceb10f20816 nodeName:}" failed. No retries permitted until 2026-09-08 02:15:37.219916093 +0000 UTC m=+1018.397228158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls") pod "maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" (UID: "a5597d72-86a8-4144-b8c7-dceb10f20816") : secret "maas-api-e2e-mt-8fa74948-serving-cert" not found Sep 08 02:15:36.720338 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.720318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.721974 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.721955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:36.728651 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:36.728630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ps6\" (UniqueName: \"kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:37.223259 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:37.223222 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:37.225956 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:37.225924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") pod \"maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:37.428230 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:37.428196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:37.705154 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:37.705124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:15:37.706509 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:15:37.706486 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5597d72_86a8_4144_b8c7_dceb10f20816.slice/crio-121bd3b0c2b32a489a56bfd15de0d8969b88caacaaeb8e3661df4e7a55ccf7b1 WatchSource:0}: Error finding container 121bd3b0c2b32a489a56bfd15de0d8969b88caacaaeb8e3661df4e7a55ccf7b1: Status 404 returned error can't find the container with id 121bd3b0c2b32a489a56bfd15de0d8969b88caacaaeb8e3661df4e7a55ccf7b1 Sep 08 02:15:38.068004 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:38.067966 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" event={"ID":"a5597d72-86a8-4144-b8c7-dceb10f20816","Type":"ContainerStarted","Data":"121bd3b0c2b32a489a56bfd15de0d8969b88caacaaeb8e3661df4e7a55ccf7b1"} Sep 08 02:15:39.075666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:39.075629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" event={"ID":"a5597d72-86a8-4144-b8c7-dceb10f20816","Type":"ContainerStarted","Data":"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369"} Sep 08 02:15:39.076124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:39.075730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:39.093165 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:39.093112 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" podStartSLOduration=2.714297814 podStartE2EDuration="3.093097549s" totalImagesPullingTime="378.799735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:15:39.090900394 +0000 UTC m=+1020.268212466" watchObservedRunningTime="2026-09-08 02:15:39.093097549 +0000 UTC m=+1020.270409621" Sep 08 02:15:45.088107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:45.088074 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:15:46.999646 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:46.999615 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:15:48.298033 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:48.297999 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:48.298525 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:48.298214 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" podUID="5379ee74-b22b-4bc1-9700-cd0caec621d4" containerName="istio-proxy" containerID="cri-o://64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b" gracePeriod=30 Sep 08 02:15:53.557823 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.557796 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:53.679539 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679507 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679704 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679544 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679704 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679582 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679704 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679634 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679704 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679682 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679909 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679909 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679909 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.679909 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679846 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgcb\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb\") pod \"5379ee74-b22b-4bc1-9700-cd0caec621d4\" (UID: \"5379ee74-b22b-4bc1-9700-cd0caec621d4\") " Sep 08 02:15:53.680124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679944 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:15:53.680124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680034 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:15:53.680124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.679961 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:15:53.680317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680232 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:15:53.680317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680247 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5379ee74-b22b-4bc1-9700-cd0caec621d4-istiod-ca-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.680317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680267 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-credential-socket\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.680317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680282 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-socket\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.680317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.680246 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:15:53.682005 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.681980 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 02:15:53.682124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.682103 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb" (OuterVolumeSpecName: "kube-api-access-jvgcb") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "kube-api-access-jvgcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:15:53.682278 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.682255 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:15:53.682278 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.682257 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5379ee74-b22b-4bc1-9700-cd0caec621d4" (UID: "5379ee74-b22b-4bc1-9700-cd0caec621d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:15:53.781586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781549 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-workload-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.781586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781580 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-data\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.781586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781589 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgcb\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-kube-api-access-jvgcb\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.781830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781600 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-envoy\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.781830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781608 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-podinfo\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:53.781830 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:53.781618 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5379ee74-b22b-4bc1-9700-cd0caec621d4-istio-token\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:15:54.171588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.171553 2559 generic.go:356] "Generic (PLEG): container finished" podID="5379ee74-b22b-4bc1-9700-cd0caec621d4" containerID="64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b" exitCode=0 Sep 08 02:15:54.171773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.171594 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" Sep 08 02:15:54.171773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.171641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" event={"ID":"5379ee74-b22b-4bc1-9700-cd0caec621d4","Type":"ContainerDied","Data":"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b"} Sep 08 02:15:54.171773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.171693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs" event={"ID":"5379ee74-b22b-4bc1-9700-cd0caec621d4","Type":"ContainerDied","Data":"8e39d0deae961ed73d9c7f3185ca842f407f041a2659f3ccaaf8712d08ec93fc"} Sep 08 02:15:54.171773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.171715 2559 scope.go:117] "RemoveContainer" containerID="64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b" Sep 08 02:15:54.182095 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.182076 2559 scope.go:117] "RemoveContainer" containerID="64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b" Sep 08 02:15:54.182451 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:15:54.182432 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b\": container with ID starting with 64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b not found: ID does not exist" containerID="64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b" Sep 08 02:15:54.182507 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.182463 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b"} err="failed to get container status \"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b\": rpc error: code = NotFound desc = could not find container \"64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b\": container with ID starting with 64601b4c20f44093dc7d9943d56d5968b16e0109e392fb3a6579b3fb5475772b not found: ID does not exist" Sep 08 02:15:54.194053 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.194021 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:54.201096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:54.201053 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-cf06cb-openshift-default-6984f4475b-65ljs"] Sep 08 02:15:55.320037 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:55.320003 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5379ee74-b22b-4bc1-9700-cd0caec621d4" path="/var/lib/kubelet/pods/5379ee74-b22b-4bc1-9700-cd0caec621d4/volumes" Sep 08 02:15:59.893975 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:59.893937 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:15:59.894458 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:15:59.894175 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" podUID="91a67d75-3523-4125-9550-dc0f65a2ed51" containerName="maas-api" containerID="cri-o://70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec" gracePeriod=30 Sep 08 02:16:00.365478 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.365454 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:16:00.446781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.446703 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") pod \"91a67d75-3523-4125-9550-dc0f65a2ed51\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " Sep 08 02:16:00.446781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.446744 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca\") pod \"91a67d75-3523-4125-9550-dc0f65a2ed51\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " Sep 08 02:16:00.446781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.446761 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs\") pod \"91a67d75-3523-4125-9550-dc0f65a2ed51\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " Sep 08 02:16:00.447032 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.446820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqvmz\" (UniqueName: \"kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz\") pod \"91a67d75-3523-4125-9550-dc0f65a2ed51\" (UID: \"91a67d75-3523-4125-9550-dc0f65a2ed51\") " Sep 08 02:16:00.447167 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.447102 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91a67d75-3523-4125-9550-dc0f65a2ed51" (UID: "91a67d75-3523-4125-9550-dc0f65a2ed51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:16:00.447798 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.447544 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a67d75-3523-4125-9550-dc0f65a2ed51-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:00.449052 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.449029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz" (OuterVolumeSpecName: "kube-api-access-wqvmz") pod "91a67d75-3523-4125-9550-dc0f65a2ed51" (UID: "91a67d75-3523-4125-9550-dc0f65a2ed51"). InnerVolumeSpecName "kube-api-access-wqvmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:16:00.449407 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.449392 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91a67d75-3523-4125-9550-dc0f65a2ed51" (UID: "91a67d75-3523-4125-9550-dc0f65a2ed51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:00.449464 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.449415 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91a67d75-3523-4125-9550-dc0f65a2ed51" (UID: "91a67d75-3523-4125-9550-dc0f65a2ed51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:00.548490 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.548456 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:00.548490 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.548487 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a67d75-3523-4125-9550-dc0f65a2ed51-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:00.548677 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:00.548500 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvmz\" (UniqueName: \"kubernetes.io/projected/91a67d75-3523-4125-9550-dc0f65a2ed51-kube-api-access-wqvmz\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:01.215306 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.215249 2559 generic.go:356] "Generic (PLEG): container finished" podID="91a67d75-3523-4125-9550-dc0f65a2ed51" containerID="70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec" exitCode=0 Sep 08 02:16:01.215306 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.215305 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" Sep 08 02:16:01.215814 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.215338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" event={"ID":"91a67d75-3523-4125-9550-dc0f65a2ed51","Type":"ContainerDied","Data":"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec"} Sep 08 02:16:01.215814 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.215407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh" event={"ID":"91a67d75-3523-4125-9550-dc0f65a2ed51","Type":"ContainerDied","Data":"d89658d550d182e62cd232a3e89b1f947e86090c9ac8efd42d16815552de0625"} Sep 08 02:16:01.215814 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.215421 2559 scope.go:117] "RemoveContainer" containerID="70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec" Sep 08 02:16:01.225961 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.225939 2559 scope.go:117] "RemoveContainer" containerID="70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec" Sep 08 02:16:01.226260 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:01.226239 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec\": container with ID starting with 70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec not found: ID does not exist" containerID="70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec" Sep 08 02:16:01.226357 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.226269 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec"} err="failed to get container status \"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec\": rpc error: code = NotFound desc = could not find container \"70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec\": container with ID starting with 70ff947c1fab7bd31257ec0092f44c0da511b5df92efccb8d43bd1f7654f83ec not found: ID does not exist" Sep 08 02:16:01.242876 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.242844 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:16:01.245318 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.245279 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d27de-56bd5ccb65-64fbh"] Sep 08 02:16:01.320274 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:01.320237 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a67d75-3523-4125-9550-dc0f65a2ed51" path="/var/lib/kubelet/pods/91a67d75-3523-4125-9550-dc0f65a2ed51/volumes" Sep 08 02:16:02.020280 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.020239 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" podUID="976e5e1c-683e-49d3-9f03-110f87a8e634" containerName="main" containerID="cri-o://7774fa24cd9c7a8573456ba5cef5812e71d4d460ee9ea06394471728e85c00a0" gracePeriod=45 Sep 08 02:16:02.223086 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.223056 2559 generic.go:356] "Generic (PLEG): container finished" podID="976e5e1c-683e-49d3-9f03-110f87a8e634" containerID="7774fa24cd9c7a8573456ba5cef5812e71d4d460ee9ea06394471728e85c00a0" exitCode=0 Sep 08 02:16:02.223602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.223129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" event={"ID":"976e5e1c-683e-49d3-9f03-110f87a8e634","Type":"ContainerDied","Data":"7774fa24cd9c7a8573456ba5cef5812e71d4d460ee9ea06394471728e85c00a0"} Sep 08 02:16:02.281783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.281719 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:16:02.365319 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365251 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365356 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365404 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365433 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5429w\" (UniqueName: \"kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365462 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365482 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache\") pod \"976e5e1c-683e-49d3-9f03-110f87a8e634\" (UID: \"976e5e1c-683e-49d3-9f03-110f87a8e634\") " Sep 08 02:16:02.365759 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home" (OuterVolumeSpecName: "home") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:16:02.365759 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365622 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:16:02.365870 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365779 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache" (OuterVolumeSpecName: "model-cache") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:16:02.365870 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365830 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-home\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:02.365870 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.365848 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-tmp-dir\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:02.367763 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.367738 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:02.367877 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.367768 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w" (OuterVolumeSpecName: "kube-api-access-5429w") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "kube-api-access-5429w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:16:02.367877 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.367771 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm" (OuterVolumeSpecName: "dshm") pod "976e5e1c-683e-49d3-9f03-110f87a8e634" (UID: "976e5e1c-683e-49d3-9f03-110f87a8e634"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:16:02.467168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.467133 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976e5e1c-683e-49d3-9f03-110f87a8e634-tls-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:02.467168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.467163 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5429w\" (UniqueName: \"kubernetes.io/projected/976e5e1c-683e-49d3-9f03-110f87a8e634-kube-api-access-5429w\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:02.467168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.467175 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-dshm\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:02.467453 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:02.467184 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976e5e1c-683e-49d3-9f03-110f87a8e634-model-cache\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:03.248958 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.248927 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" Sep 08 02:16:03.249432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.248953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj" event={"ID":"976e5e1c-683e-49d3-9f03-110f87a8e634","Type":"ContainerDied","Data":"57bdfa7201e4c370b631696189568b426c6b9ca7d54b169c96f233a20fb524c0"} Sep 08 02:16:03.249432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.249014 2559 scope.go:117] "RemoveContainer" containerID="7774fa24cd9c7a8573456ba5cef5812e71d4d460ee9ea06394471728e85c00a0" Sep 08 02:16:03.271369 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.271337 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:16:03.274813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.274783 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5d27de/test-model-5d27de-kserve-59d4f58475-b55jj"] Sep 08 02:16:03.320781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:03.320738 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976e5e1c-683e-49d3-9f03-110f87a8e634" path="/var/lib/kubelet/pods/976e5e1c-683e-49d3-9f03-110f87a8e634/volumes" Sep 08 02:16:17.372556 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.372518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:16:17.373076 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.372731 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerName="maas-api" containerID="cri-o://3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200" gracePeriod=30 Sep 08 02:16:17.626991 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.626937 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:16:17.694098 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca\") pod \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " Sep 08 02:16:17.694267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694149 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjdkd\" (UniqueName: \"kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd\") pod \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " Sep 08 02:16:17.694267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls\") pod \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " Sep 08 02:16:17.694267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694191 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs\") pod \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\" (UID: \"5b364411-3dce-4b3a-a0f3-58dc6d726d8f\") " Sep 08 02:16:17.694471 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694351 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b364411-3dce-4b3a-a0f3-58dc6d726d8f" (UID: "5b364411-3dce-4b3a-a0f3-58dc6d726d8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:16:17.694535 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.694498 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:17.696221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.696195 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b364411-3dce-4b3a-a0f3-58dc6d726d8f" (UID: "5b364411-3dce-4b3a-a0f3-58dc6d726d8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:17.696221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.696211 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b364411-3dce-4b3a-a0f3-58dc6d726d8f" (UID: "5b364411-3dce-4b3a-a0f3-58dc6d726d8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:17.696356 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.696322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd" (OuterVolumeSpecName: "kube-api-access-tjdkd") pod "5b364411-3dce-4b3a-a0f3-58dc6d726d8f" (UID: "5b364411-3dce-4b3a-a0f3-58dc6d726d8f"). InnerVolumeSpecName "kube-api-access-tjdkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:16:17.795255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.795217 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdkd\" (UniqueName: \"kubernetes.io/projected/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-kube-api-access-tjdkd\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:17.795255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.795250 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:17.795255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:17.795262 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b364411-3dce-4b3a-a0f3-58dc6d726d8f-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:18.343461 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.343422 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerID="3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200" exitCode=0 Sep 08 02:16:18.343709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.343476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" Sep 08 02:16:18.343709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.343512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" event={"ID":"5b364411-3dce-4b3a-a0f3-58dc6d726d8f","Type":"ContainerDied","Data":"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200"} Sep 08 02:16:18.343709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.343592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" event={"ID":"5b364411-3dce-4b3a-a0f3-58dc6d726d8f","Type":"ContainerDied","Data":"053359e159da9f477fba8f5ac228a7f6aef1ac035a9554bfba1a56199f104f90"} Sep 08 02:16:18.343709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.343616 2559 scope.go:117] "RemoveContainer" containerID="3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200" Sep 08 02:16:18.361975 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.361949 2559 scope.go:117] "RemoveContainer" containerID="3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200" Sep 08 02:16:18.362386 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:18.362358 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200\": container with ID starting with 3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200 not found: ID does not exist" containerID="3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200" Sep 08 02:16:18.362456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.362402 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200"} err="failed to get container status \"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200\": rpc error: code = NotFound desc = could not find container \"3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200\": container with ID starting with 3913833c153649b9bbb6eb240de4d52f0f5df0bb23a1706f8061cdcf581e2200 not found: ID does not exist" Sep 08 02:16:18.375104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.375074 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:16:18.381246 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.381218 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66"] Sep 08 02:16:18.579395 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.579358 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:16:18.579617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.579590 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" podUID="a5597d72-86a8-4144-b8c7-dceb10f20816" containerName="maas-api" containerID="cri-o://08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369" gracePeriod=30 Sep 08 02:16:18.835611 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.835582 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:16:18.903358 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.903327 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") pod \"a5597d72-86a8-4144-b8c7-dceb10f20816\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " Sep 08 02:16:18.903512 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.903441 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2ps6\" (UniqueName: \"kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6\") pod \"a5597d72-86a8-4144-b8c7-dceb10f20816\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " Sep 08 02:16:18.903567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.903512 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca\") pod \"a5597d72-86a8-4144-b8c7-dceb10f20816\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " Sep 08 02:16:18.903610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.903569 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs\") pod \"a5597d72-86a8-4144-b8c7-dceb10f20816\" (UID: \"a5597d72-86a8-4144-b8c7-dceb10f20816\") " Sep 08 02:16:18.903894 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.903867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5597d72-86a8-4144-b8c7-dceb10f20816" (UID: "a5597d72-86a8-4144-b8c7-dceb10f20816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:16:18.905416 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.905382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5597d72-86a8-4144-b8c7-dceb10f20816" (UID: "a5597d72-86a8-4144-b8c7-dceb10f20816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:18.905609 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.905587 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5597d72-86a8-4144-b8c7-dceb10f20816" (UID: "a5597d72-86a8-4144-b8c7-dceb10f20816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:16:18.905690 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:18.905657 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6" (OuterVolumeSpecName: "kube-api-access-l2ps6") pod "a5597d72-86a8-4144-b8c7-dceb10f20816" (UID: "a5597d72-86a8-4144-b8c7-dceb10f20816"). InnerVolumeSpecName "kube-api-access-l2ps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:16:19.005129 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.005094 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:19.005129 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.005125 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5597d72-86a8-4144-b8c7-dceb10f20816-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:19.005129 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.005134 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ps6\" (UniqueName: \"kubernetes.io/projected/a5597d72-86a8-4144-b8c7-dceb10f20816-kube-api-access-l2ps6\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:19.005379 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.005143 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5597d72-86a8-4144-b8c7-dceb10f20816-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:16:19.319683 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.319605 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" path="/var/lib/kubelet/pods/5b364411-3dce-4b3a-a0f3-58dc6d726d8f/volumes" Sep 08 02:16:19.351519 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.351483 2559 generic.go:356] "Generic (PLEG): container finished" podID="a5597d72-86a8-4144-b8c7-dceb10f20816" containerID="08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369" exitCode=0 Sep 08 02:16:19.351664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.351529 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" Sep 08 02:16:19.351664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.351569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" event={"ID":"a5597d72-86a8-4144-b8c7-dceb10f20816","Type":"ContainerDied","Data":"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369"} Sep 08 02:16:19.351664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.351622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5" event={"ID":"a5597d72-86a8-4144-b8c7-dceb10f20816","Type":"ContainerDied","Data":"121bd3b0c2b32a489a56bfd15de0d8969b88caacaaeb8e3661df4e7a55ccf7b1"} Sep 08 02:16:19.351664 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.351647 2559 scope.go:117] "RemoveContainer" containerID="08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369" Sep 08 02:16:19.362727 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.362705 2559 scope.go:117] "RemoveContainer" containerID="08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369" Sep 08 02:16:19.363025 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:19.362994 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369\": container with ID starting with 08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369 not found: ID does not exist" containerID="08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369" Sep 08 02:16:19.363132 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.363023 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369"} err="failed to get container status \"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369\": rpc error: code = NotFound desc = could not find container \"08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369\": container with ID starting with 08e2d6686c0eef6039971312ca44023a161e80127641e87e43cbb73459260369 not found: ID does not exist" Sep 08 02:16:19.369265 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.369240 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:16:19.371654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:19.371631 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fa74948-75585dccd5-jwcm5"] Sep 08 02:16:20.619794 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:20.619752 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d800a-789ffd484b-dfh66" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": context deadline exceeded" Sep 08 02:16:21.319644 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:21.319613 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5597d72-86a8-4144-b8c7-dceb10f20816" path="/var/lib/kubelet/pods/a5597d72-86a8-4144-b8c7-dceb10f20816/volumes" Sep 08 02:16:24.946307 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946263 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946671 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5597d72-86a8-4144-b8c7-dceb10f20816" containerName="maas-api" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946684 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5597d72-86a8-4144-b8c7-dceb10f20816" containerName="maas-api" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946699 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5379ee74-b22b-4bc1-9700-cd0caec621d4" containerName="istio-proxy" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946705 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5379ee74-b22b-4bc1-9700-cd0caec621d4" containerName="istio-proxy" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946716 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerName="maas-api" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946722 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerName="maas-api" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946733 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976e5e1c-683e-49d3-9f03-110f87a8e634" containerName="main" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946738 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="976e5e1c-683e-49d3-9f03-110f87a8e634" containerName="main" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946750 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a67d75-3523-4125-9550-dc0f65a2ed51" containerName="maas-api" Sep 08 02:16:24.946757 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946754 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a67d75-3523-4125-9550-dc0f65a2ed51" containerName="maas-api" Sep 08 02:16:24.947100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946809 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="976e5e1c-683e-49d3-9f03-110f87a8e634" containerName="main" Sep 08 02:16:24.947100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946817 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b364411-3dce-4b3a-a0f3-58dc6d726d8f" containerName="maas-api" Sep 08 02:16:24.947100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946824 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5597d72-86a8-4144-b8c7-dceb10f20816" containerName="maas-api" Sep 08 02:16:24.947100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946830 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5379ee74-b22b-4bc1-9700-cd0caec621d4" containerName="istio-proxy" Sep 08 02:16:24.947100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.946835 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a67d75-3523-4125-9550-dc0f65a2ed51" containerName="maas-api" Sep 08 02:16:24.951014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.950995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:24.953199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.953177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 02:16:24.953347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.953306 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-fcdd46-openshift-default-dockercfg-tsb7x\"" Sep 08 02:16:24.962347 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:24.962325 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:16:25.058450 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ggr\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058450 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.058972 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.058753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.159931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.159894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ggr\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.159931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.159935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.159950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160451 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160426 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160670 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.160890 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.161021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.161021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.160918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.162350 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.162331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.162943 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.162928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.171856 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.171832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.172019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.171996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ggr\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr\") pod \"e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.262335 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.262216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:25.540025 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.540000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:16:25.542817 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:16:25.542780 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7856e7b2_150e_4979_97f0_6fdcd2b3f1c2.slice/crio-a24c0e64030905c32147d491ae22f152a62d57811c32dab1d6a75133e4748e0a WatchSource:0}: Error finding container a24c0e64030905c32147d491ae22f152a62d57811c32dab1d6a75133e4748e0a: Status 404 returned error can't find the container with id a24c0e64030905c32147d491ae22f152a62d57811c32dab1d6a75133e4748e0a Sep 08 02:16:25.545109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.545079 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:25.545193 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.545143 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:25.545193 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:25.545170 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:26.396491 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:26.396448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerStarted","Data":"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90"} Sep 08 02:16:26.396967 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:26.396511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerStarted","Data":"a24c0e64030905c32147d491ae22f152a62d57811c32dab1d6a75133e4748e0a"} Sep 08 02:16:26.419507 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:26.419460 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podStartSLOduration=2.419442993 podStartE2EDuration="2.419442993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:26.418400194 +0000 UTC m=+1067.595712289" watchObservedRunningTime="2026-09-08 02:16:26.419442993 +0000 UTC m=+1067.596755064" Sep 08 02:16:27.262601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:27.262562 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:27.269179 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:27.269152 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:27.402762 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:27.402729 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:27.403660 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:27.403640 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:29.407736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.407700 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:16:29.411638 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.411606 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.413705 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.413673 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-fcdd46-serving-cert\"" Sep 08 02:16:29.422792 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.422764 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:16:29.502696 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.502660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.502913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.502728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkkg\" (UniqueName: \"kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.502913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.502812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.503062 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.502930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.603400 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.603363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.603597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.603422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.603597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.603462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkkg\" (UniqueName: \"kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.603597 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.603517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.603774 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:29.603520 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-serving-cert: secret "maas-api-e2e-shared-a-w3-fcdd46-serving-cert" not found Sep 08 02:16:29.603774 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:29.603665 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls podName:0899785a-959b-4612-b214-addb714eec98 nodeName:}" failed. No retries permitted until 2026-09-08 02:16:30.10364316 +0000 UTC m=+1071.280955213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls") pod "maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" (UID: "0899785a-959b-4612-b214-addb714eec98") : secret "maas-api-e2e-shared-a-w3-fcdd46-serving-cert" not found Sep 08 02:16:29.604343 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.604318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.605839 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.605822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:29.613269 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:29.613248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkkg\" (UniqueName: \"kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:30.108993 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:30.108954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:30.111449 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:30.111428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:30.324821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:30.324781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:30.604702 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:30.604662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:16:30.607001 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:16:30.606963 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0899785a_959b_4612_b214_addb714eec98.slice/crio-adc76d4c75b9759a4f958ee0cfe109b2d6bc259415145ee5e699a42a24936ae3 WatchSource:0}: Error finding container adc76d4c75b9759a4f958ee0cfe109b2d6bc259415145ee5e699a42a24936ae3: Status 404 returned error can't find the container with id adc76d4c75b9759a4f958ee0cfe109b2d6bc259415145ee5e699a42a24936ae3 Sep 08 02:16:30.608405 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:30.608385 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 02:16:31.428867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:31.428830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" event={"ID":"0899785a-959b-4612-b214-addb714eec98","Type":"ContainerStarted","Data":"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d"} Sep 08 02:16:31.429061 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:31.428880 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" event={"ID":"0899785a-959b-4612-b214-addb714eec98","Type":"ContainerStarted","Data":"adc76d4c75b9759a4f958ee0cfe109b2d6bc259415145ee5e699a42a24936ae3"} Sep 08 02:16:31.429061 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:31.428943 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:31.445699 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:31.445643 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" podStartSLOduration=2.097989465 podStartE2EDuration="2.44562326s" totalImagesPullingTime="347.633795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:31.445067186 +0000 UTC m=+1072.622379257" watchObservedRunningTime="2026-09-08 02:16:31.44562326 +0000 UTC m=+1072.622935336" Sep 08 02:16:37.440115 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:37.440082 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:16:42.766851 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:42.766134 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": read tcp 10.132.0.2:60874->10.132.0.82:15021: read: connection reset by peer" start-of-body= Sep 08 02:16:42.766851 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:42.766215 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": read tcp 10.132.0.2:60874->10.132.0.82:15021: read: connection reset by peer" Sep 08 02:16:43.513331 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.513275 2559 generic.go:356] "Generic (PLEG): container finished" podID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerID="7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90" exitCode=0 Sep 08 02:16:43.513511 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.513359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerDied","Data":"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90"} Sep 08 02:16:43.513838 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.513822 2559 scope.go:117] "RemoveContainer" containerID="7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90" Sep 08 02:16:43.514856 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.514827 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:43.514955 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.514911 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:43.515018 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.514959 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 02:16:43.911772 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.911737 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:16:43.915504 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.915486 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:43.917719 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.917698 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-670bc3-serving-cert\"" Sep 08 02:16:43.926345 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:43.926318 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:16:44.044104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.044052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.044104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.044096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgr4\" (UniqueName: \"kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.044360 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.044120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.044401 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.044347 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.145716 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.145675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.145904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.145750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.145904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.145769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgr4\" (UniqueName: \"kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.145904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.145787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.145904 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:44.145855 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-serving-cert: secret "maas-api-e2e-shared-a-w2-670bc3-serving-cert" not found Sep 08 02:16:44.146070 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:16:44.145943 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls podName:60710743-598c-4821-aa9a-67917db6fd52 nodeName:}" failed. No retries permitted until 2026-09-08 02:16:44.645921127 +0000 UTC m=+1085.823233186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls") pod "maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" (UID: "60710743-598c-4821-aa9a-67917db6fd52") : secret "maas-api-e2e-shared-a-w2-670bc3-serving-cert" not found Sep 08 02:16:44.146479 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.146461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.148315 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.148280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.153977 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.153951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgr4\" (UniqueName: \"kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.521644 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.521608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerStarted","Data":"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1"} Sep 08 02:16:44.650844 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.650802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.653197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.653172 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:44.827158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:44.827068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:45.124055 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:45.124021 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:16:45.125650 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:16:45.125625 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60710743_598c_4821_aa9a_67917db6fd52.slice/crio-589a4384688eddfdf8ec7b8180c18158804ea6bbb9de249d9c0fd1365cef0716 WatchSource:0}: Error finding container 589a4384688eddfdf8ec7b8180c18158804ea6bbb9de249d9c0fd1365cef0716: Status 404 returned error can't find the container with id 589a4384688eddfdf8ec7b8180c18158804ea6bbb9de249d9c0fd1365cef0716 Sep 08 02:16:45.262949 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:45.262912 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:45.263163 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:45.263133 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 08 02:16:45.263238 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:45.263200 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 08 02:16:45.528709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:45.528669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" event={"ID":"60710743-598c-4821-aa9a-67917db6fd52","Type":"ContainerStarted","Data":"589a4384688eddfdf8ec7b8180c18158804ea6bbb9de249d9c0fd1365cef0716"} Sep 08 02:16:46.262971 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:46.262926 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 08 02:16:46.263516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:46.262999 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 08 02:16:46.536645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:46.536553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" event={"ID":"60710743-598c-4821-aa9a-67917db6fd52","Type":"ContainerStarted","Data":"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa"} Sep 08 02:16:46.536645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:46.536637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:46.553358 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:46.553285 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" podStartSLOduration=3.097593579 podStartE2EDuration="3.553269775s" totalImagesPullingTime="455.676196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:46.55166424 +0000 UTC m=+1087.728976337" watchObservedRunningTime="2026-09-08 02:16:46.553269775 +0000 UTC m=+1087.730581848" Sep 08 02:16:47.263802 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:47.263766 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 08 02:16:47.264801 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:47.264399 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 08 02:16:48.263395 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:48.263352 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 08 02:16:48.263587 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:48.263416 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 08 02:16:49.263397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:49.262944 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 08 02:16:49.263397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:49.263005 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 08 02:16:50.266971 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:50.266939 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:50.267446 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:50.267324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:50.268210 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:50.268189 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:16:52.547868 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:52.547837 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:16:57.917183 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.917146 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:16:57.921684 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.921663 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:57.923763 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.923739 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-527ac8-serving-cert\"" Sep 08 02:16:57.929670 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.929650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:16:57.987342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.987315 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:57.987342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.987344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:57.987534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.987363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:57.987534 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:57.987447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.088719 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.088692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.088874 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.088779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.088874 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.088804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.088874 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.088821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.089488 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.089463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.091087 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.091063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.091211 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.091193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.102588 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.102569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j\") pod \"maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.234204 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.234135 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:58.509300 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.509254 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:16:58.511127 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:16:58.511105 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa4997_0ede_4470_bf6f_15042f79f152.slice/crio-0066d0bc594c289b7501142a6699d2943e0383307884a6d67e63cb1ef0618318 WatchSource:0}: Error finding container 0066d0bc594c289b7501142a6699d2943e0383307884a6d67e63cb1ef0618318: Status 404 returned error can't find the container with id 0066d0bc594c289b7501142a6699d2943e0383307884a6d67e63cb1ef0618318 Sep 08 02:16:58.612317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:58.612256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" event={"ID":"82fa4997-0ede-4470-bf6f-15042f79f152","Type":"ContainerStarted","Data":"0066d0bc594c289b7501142a6699d2943e0383307884a6d67e63cb1ef0618318"} Sep 08 02:16:59.619976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:59.619938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" event={"ID":"82fa4997-0ede-4470-bf6f-15042f79f152","Type":"ContainerStarted","Data":"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927"} Sep 08 02:16:59.620389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:59.620018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:16:59.636001 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:16:59.635952 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" podStartSLOduration=2.297727929 podStartE2EDuration="2.635935484s" totalImagesPullingTime="338.207555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:59.634524204 +0000 UTC m=+1100.811836280" watchObservedRunningTime="2026-09-08 02:16:59.635935484 +0000 UTC m=+1100.813247589" Sep 08 02:17:05.632027 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:05.631999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:17:12.040881 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.040844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:17:12.045059 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.045034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.047886 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.047864 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-1d1413-serving-cert\"" Sep 08 02:17:12.058960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.058926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:17:12.134343 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.134313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.134343 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.134348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.134591 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.134441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngqg\" (UniqueName: \"kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.134591 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.134470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.235683 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.235645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.235871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.235806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.235871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.235839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.235993 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.235929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngqg\" (UniqueName: \"kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.236589 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.236558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.238206 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.238177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.238332 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.238219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.243921 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.243902 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngqg\" (UniqueName: \"kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg\") pod \"maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.358895 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.358807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:12.637081 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.637054 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:17:12.638478 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:12.638447 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617aabf7_df68_4db3_8f77_3574821cb4df.slice/crio-52453d2ce81010420a55db0d7b3fa048199cb9236bb6a367611baceb9a9e75b2 WatchSource:0}: Error finding container 52453d2ce81010420a55db0d7b3fa048199cb9236bb6a367611baceb9a9e75b2: Status 404 returned error can't find the container with id 52453d2ce81010420a55db0d7b3fa048199cb9236bb6a367611baceb9a9e75b2 Sep 08 02:17:12.701592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:12.701558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" event={"ID":"617aabf7-df68-4db3-8f77-3574821cb4df","Type":"ContainerStarted","Data":"52453d2ce81010420a55db0d7b3fa048199cb9236bb6a367611baceb9a9e75b2"} Sep 08 02:17:13.708614 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:13.708572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" event={"ID":"617aabf7-df68-4db3-8f77-3574821cb4df","Type":"ContainerStarted","Data":"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254"} Sep 08 02:17:13.709000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:13.708686 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:13.727517 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:13.727470 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" podStartSLOduration=1.350563061 podStartE2EDuration="1.727454924s" totalImagesPullingTime="376.891863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:13.724983939 +0000 UTC m=+1114.902296012" watchObservedRunningTime="2026-09-08 02:17:13.727454924 +0000 UTC m=+1114.904766996" Sep 08 02:17:19.720949 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:19.720916 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:17:36.484299 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.484253 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:36.489273 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.489246 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:36.493212 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.493119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:36.494307 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.494275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2574n\"" Sep 08 02:17:36.668448 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.668409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdrn\" (UniqueName: \"kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn\") pod \"test-curl-7679-139746767632112\" (UID: \"5ab2a01b-fd19-4a70-817c-c01ebfb0e620\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:36.769685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.769595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdrn\" (UniqueName: \"kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn\") pod \"test-curl-7679-139746767632112\" (UID: \"5ab2a01b-fd19-4a70-817c-c01ebfb0e620\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:36.777697 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.777662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdrn\" (UniqueName: \"kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn\") pod \"test-curl-7679-139746767632112\" (UID: \"5ab2a01b-fd19-4a70-817c-c01ebfb0e620\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:36.801623 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:36.801591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:37.077476 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:37.077446 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:37.079349 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:37.079318 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ab2a01b_fd19_4a70_817c_c01ebfb0e620.slice/crio-01636dcb2bb05dacb64d4391fa27a98e153412a5b7d0a5bdfa1dfed1986f5f3e WatchSource:0}: Error finding container 01636dcb2bb05dacb64d4391fa27a98e153412a5b7d0a5bdfa1dfed1986f5f3e: Status 404 returned error can't find the container with id 01636dcb2bb05dacb64d4391fa27a98e153412a5b7d0a5bdfa1dfed1986f5f3e Sep 08 02:17:37.857211 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:37.857183 2559 generic.go:356] "Generic (PLEG): container finished" podID="5ab2a01b-fd19-4a70-817c-c01ebfb0e620" containerID="18925cd9bd5cff29cd75405a86c7a186633ca326f3592f2f8469ac068525ff9a" exitCode=0 Sep 08 02:17:37.857625 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:37.857260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632112" event={"ID":"5ab2a01b-fd19-4a70-817c-c01ebfb0e620","Type":"ContainerDied","Data":"18925cd9bd5cff29cd75405a86c7a186633ca326f3592f2f8469ac068525ff9a"} Sep 08 02:17:37.857625 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:37.857325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632112" event={"ID":"5ab2a01b-fd19-4a70-817c-c01ebfb0e620","Type":"ContainerStarted","Data":"01636dcb2bb05dacb64d4391fa27a98e153412a5b7d0a5bdfa1dfed1986f5f3e"} Sep 08 02:17:39.004948 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.004927 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.035794 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.035761 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-139746767632112_5ab2a01b-fd19-4a70-817c-c01ebfb0e620/test-curl-7679-139746767632112/0.log" Sep 08 02:17:39.057593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.057563 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:39.061750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.061721 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:39.103263 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.103239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gdrn\" (UniqueName: \"kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn\") pod \"5ab2a01b-fd19-4a70-817c-c01ebfb0e620\" (UID: \"5ab2a01b-fd19-4a70-817c-c01ebfb0e620\") " Sep 08 02:17:39.105208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.105183 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn" (OuterVolumeSpecName: "kube-api-access-6gdrn") pod "5ab2a01b-fd19-4a70-817c-c01ebfb0e620" (UID: "5ab2a01b-fd19-4a70-817c-c01ebfb0e620"). InnerVolumeSpecName "kube-api-access-6gdrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:17:39.172681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.172657 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:39.173109 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.173096 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab2a01b-fd19-4a70-817c-c01ebfb0e620" containerName="test-curl-7679-139746767632112" Sep 08 02:17:39.173158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.173110 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab2a01b-fd19-4a70-817c-c01ebfb0e620" containerName="test-curl-7679-139746767632112" Sep 08 02:17:39.173194 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.173189 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab2a01b-fd19-4a70-817c-c01ebfb0e620" containerName="test-curl-7679-139746767632112" Sep 08 02:17:39.176343 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.176324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.181860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.181838 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:39.204402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.204378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfw8\" (UniqueName: \"kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8\") pod \"test-curl-7679-139746767632112\" (UID: \"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.204553 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.204535 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdrn\" (UniqueName: \"kubernetes.io/projected/5ab2a01b-fd19-4a70-817c-c01ebfb0e620-kube-api-access-6gdrn\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:17:39.306012 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.305967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfw8\" (UniqueName: \"kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8\") pod \"test-curl-7679-139746767632112\" (UID: \"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.319457 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.319428 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab2a01b-fd19-4a70-817c-c01ebfb0e620" path="/var/lib/kubelet/pods/5ab2a01b-fd19-4a70-817c-c01ebfb0e620/volumes" Sep 08 02:17:39.331518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.331492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfw8\" (UniqueName: \"kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8\") pod \"test-curl-7679-139746767632112\" (UID: \"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d\") " pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.487406 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.487331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.761245 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.761219 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:39.762993 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:39.762956 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cbdaa14_0143_4e11_a733_a1a5e2a26a7d.slice/crio-4c2685ed5b8810e542bd6af1b3110f1369554d3f16decc5141f3405bf8e7629b WatchSource:0}: Error finding container 4c2685ed5b8810e542bd6af1b3110f1369554d3f16decc5141f3405bf8e7629b: Status 404 returned error can't find the container with id 4c2685ed5b8810e542bd6af1b3110f1369554d3f16decc5141f3405bf8e7629b Sep 08 02:17:39.872570 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.872533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:39.872570 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.872552 2559 scope.go:117] "RemoveContainer" containerID="18925cd9bd5cff29cd75405a86c7a186633ca326f3592f2f8469ac068525ff9a" Sep 08 02:17:39.873899 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:39.873870 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632112" event={"ID":"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d","Type":"ContainerStarted","Data":"4c2685ed5b8810e542bd6af1b3110f1369554d3f16decc5141f3405bf8e7629b"} Sep 08 02:17:40.882205 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:40.882119 2559 generic.go:356] "Generic (PLEG): container finished" podID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" containerID="855af113854895debbb79c7efc42be06dd5bdc4d9a30e1b8286b0854e78dd3e4" exitCode=0 Sep 08 02:17:40.882611 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:40.882194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632112" event={"ID":"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d","Type":"ContainerDied","Data":"855af113854895debbb79c7efc42be06dd5bdc4d9a30e1b8286b0854e78dd3e4"} Sep 08 02:17:42.025525 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.025503 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:42.045034 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.045014 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-139746767632112_7cbdaa14-0143-4e11-a733-a1a5e2a26a7d/test-curl-7679-139746767632112/0.log" Sep 08 02:17:42.066196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.066174 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:42.069020 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.069001 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-139746767632112"] Sep 08 02:17:42.129983 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.129962 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxfw8\" (UniqueName: \"kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8\") pod \"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d\" (UID: \"7cbdaa14-0143-4e11-a733-a1a5e2a26a7d\") " Sep 08 02:17:42.131890 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.131869 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8" (OuterVolumeSpecName: "kube-api-access-dxfw8") pod "7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" (UID: "7cbdaa14-0143-4e11-a733-a1a5e2a26a7d"). InnerVolumeSpecName "kube-api-access-dxfw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:17:42.182503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.182479 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:42.182880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.182867 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" containerName="test-curl-7679-139746767632112" Sep 08 02:17:42.182932 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.182881 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" containerName="test-curl-7679-139746767632112" Sep 08 02:17:42.182979 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.182969 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" containerName="test-curl-7679-139746767632112" Sep 08 02:17:42.186074 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.186060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:42.192789 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.192763 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:42.231475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.231452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz8j\" (UniqueName: \"kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j\") pod \"test-curl-7679-139746767632688\" (UID: \"3cfb87a9-398a-4bd9-a097-9767a4f06c64\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:42.231603 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.231501 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxfw8\" (UniqueName: \"kubernetes.io/projected/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d-kube-api-access-dxfw8\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:17:42.332572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.332542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz8j\" (UniqueName: \"kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j\") pod \"test-curl-7679-139746767632688\" (UID: \"3cfb87a9-398a-4bd9-a097-9767a4f06c64\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:42.340503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.340477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz8j\" (UniqueName: \"kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j\") pod \"test-curl-7679-139746767632688\" (UID: \"3cfb87a9-398a-4bd9-a097-9767a4f06c64\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:42.496495 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.496418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:42.766997 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.766971 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:42.769570 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:42.769547 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cfb87a9_398a_4bd9_a097_9767a4f06c64.slice/crio-b0f43c369b13a46bd4462456181c3e834d021d6d36ecb3bd7bc4cdcd51dc7395 WatchSource:0}: Error finding container b0f43c369b13a46bd4462456181c3e834d021d6d36ecb3bd7bc4cdcd51dc7395: Status 404 returned error can't find the container with id b0f43c369b13a46bd4462456181c3e834d021d6d36ecb3bd7bc4cdcd51dc7395 Sep 08 02:17:42.894711 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.894675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632688" event={"ID":"3cfb87a9-398a-4bd9-a097-9767a4f06c64","Type":"ContainerStarted","Data":"b0f43c369b13a46bd4462456181c3e834d021d6d36ecb3bd7bc4cdcd51dc7395"} Sep 08 02:17:42.895849 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.895827 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:42.895849 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:42.895843 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2685ed5b8810e542bd6af1b3110f1369554d3f16decc5141f3405bf8e7629b" Sep 08 02:17:42.906062 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:17:42.906029 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-139746767632112\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" pod="opendatahub/test-curl-7679-139746767632112" Sep 08 02:17:43.320196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:43.320164 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbdaa14-0143-4e11-a733-a1a5e2a26a7d" path="/var/lib/kubelet/pods/7cbdaa14-0143-4e11-a733-a1a5e2a26a7d/volumes" Sep 08 02:17:43.904455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:43.904421 2559 generic.go:356] "Generic (PLEG): container finished" podID="3cfb87a9-398a-4bd9-a097-9767a4f06c64" containerID="34a879b04621e4bf0d0e5858603c8ef897281d1610aefbfb16ee23babe8c8340" exitCode=0 Sep 08 02:17:43.904647 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:43.904481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632688" event={"ID":"3cfb87a9-398a-4bd9-a097-9767a4f06c64","Type":"ContainerDied","Data":"34a879b04621e4bf0d0e5858603c8ef897281d1610aefbfb16ee23babe8c8340"} Sep 08 02:17:45.054115 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.054086 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.078018 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.077983 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-139746767632688_3cfb87a9-398a-4bd9-a097-9767a4f06c64/test-curl-7679-139746767632688/0.log" Sep 08 02:17:45.099806 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.099772 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:45.103889 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.103861 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:45.161650 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.161558 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzz8j\" (UniqueName: \"kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j\") pod \"3cfb87a9-398a-4bd9-a097-9767a4f06c64\" (UID: \"3cfb87a9-398a-4bd9-a097-9767a4f06c64\") " Sep 08 02:17:45.163649 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.163619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j" (OuterVolumeSpecName: "kube-api-access-dzz8j") pod "3cfb87a9-398a-4bd9-a097-9767a4f06c64" (UID: "3cfb87a9-398a-4bd9-a097-9767a4f06c64"). InnerVolumeSpecName "kube-api-access-dzz8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:17:45.219871 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.219834 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:45.220303 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.220279 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfb87a9-398a-4bd9-a097-9767a4f06c64" containerName="test-curl-7679-139746767632688" Sep 08 02:17:45.220356 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.220306 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfb87a9-398a-4bd9-a097-9767a4f06c64" containerName="test-curl-7679-139746767632688" Sep 08 02:17:45.220397 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.220388 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfb87a9-398a-4bd9-a097-9767a4f06c64" containerName="test-curl-7679-139746767632688" Sep 08 02:17:45.223611 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.223596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.231672 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.231404 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:45.262709 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.262670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclwc\" (UniqueName: \"kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc\") pod \"test-curl-7679-139746767632688\" (UID: \"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.262875 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.262772 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzz8j\" (UniqueName: \"kubernetes.io/projected/3cfb87a9-398a-4bd9-a097-9767a4f06c64-kube-api-access-dzz8j\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:17:45.320429 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.320395 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfb87a9-398a-4bd9-a097-9767a4f06c64" path="/var/lib/kubelet/pods/3cfb87a9-398a-4bd9-a097-9767a4f06c64/volumes" Sep 08 02:17:45.363565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.363532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclwc\" (UniqueName: \"kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc\") pod \"test-curl-7679-139746767632688\" (UID: \"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.377930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.377899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclwc\" (UniqueName: \"kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc\") pod \"test-curl-7679-139746767632688\" (UID: \"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23\") " pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.534452 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.534367 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.811444 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.811419 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:45.918642 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.918608 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:45.918831 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.918647 2559 scope.go:117] "RemoveContainer" containerID="34a879b04621e4bf0d0e5858603c8ef897281d1610aefbfb16ee23babe8c8340" Sep 08 02:17:45.920159 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:45.920132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632688" event={"ID":"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23","Type":"ContainerStarted","Data":"b02c4633fd579876066347dd2f0fb7d45d6bbb8adca7656550c32fdb6617e544"} Sep 08 02:17:47.934735 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:47.934700 2559 generic.go:356] "Generic (PLEG): container finished" podID="a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" containerID="ca509b74c907a1c8b462815233cbd64ad6c5706a6ac5ff48cf1d7ca775ef89cb" exitCode=0 Sep 08 02:17:47.935113 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:47.934786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-139746767632688" event={"ID":"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23","Type":"ContainerDied","Data":"ca509b74c907a1c8b462815233cbd64ad6c5706a6ac5ff48cf1d7ca775ef89cb"} Sep 08 02:17:49.079707 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.079684 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:49.097380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.097355 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-139746767632688_a6bdd804-7635-4af6-afb2-d0d9d7ca2b23/test-curl-7679-139746767632688/0.log" Sep 08 02:17:49.120587 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.120558 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:49.124768 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.124746 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-139746767632688"] Sep 08 02:17:49.200558 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.200478 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sclwc\" (UniqueName: \"kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc\") pod \"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23\" (UID: \"a6bdd804-7635-4af6-afb2-d0d9d7ca2b23\") " Sep 08 02:17:49.202510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.202487 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc" (OuterVolumeSpecName: "kube-api-access-sclwc") pod "a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" (UID: "a6bdd804-7635-4af6-afb2-d0d9d7ca2b23"). InnerVolumeSpecName "kube-api-access-sclwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:17:49.301393 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.301360 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclwc\" (UniqueName: \"kubernetes.io/projected/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23-kube-api-access-sclwc\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:17:49.319791 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.319763 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" path="/var/lib/kubelet/pods/a6bdd804-7635-4af6-afb2-d0d9d7ca2b23/volumes" Sep 08 02:17:49.948845 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.948816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-139746767632688" Sep 08 02:17:49.949062 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:49.948815 2559 scope.go:117] "RemoveContainer" containerID="ca509b74c907a1c8b462815233cbd64ad6c5706a6ac5ff48cf1d7ca775ef89cb" Sep 08 02:17:54.248941 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.248897 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:17:54.249587 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.249567 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" containerName="test-curl-7679-139746767632688" Sep 08 02:17:54.249641 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.249590 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" containerName="test-curl-7679-139746767632688" Sep 08 02:17:54.249724 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.249713 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6bdd804-7635-4af6-afb2-d0d9d7ca2b23" containerName="test-curl-7679-139746767632688" Sep 08 02:17:54.254730 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.254702 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.257400 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.257376 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-967ab780-serving-cert\"" Sep 08 02:17:54.260282 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.260260 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:17:54.347708 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.347678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthn7\" (UniqueName: \"kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.347848 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.347730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.347848 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.347830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.347942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.347876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.448701 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.448664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.448897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.448719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.448897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.448771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthn7\" (UniqueName: \"kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.448897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.448838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.449073 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:17:54.448945 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-serving-cert: secret "maas-api-e2e-mt-967ab780-serving-cert" not found Sep 08 02:17:54.449073 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:17:54.449003 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls podName:dfae7f97-54bb-46d9-945f-73651491f0c9 nodeName:}" failed. No retries permitted until 2026-09-08 02:17:54.948983285 +0000 UTC m=+1156.126295335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls") pod "maas-api-e2e-mt-967ab780-5f46456974-zqjqf" (UID: "dfae7f97-54bb-46d9-945f-73651491f0c9") : secret "maas-api-e2e-mt-967ab780-serving-cert" not found Sep 08 02:17:54.449455 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.449429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.451240 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.451220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.456540 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.456503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthn7\" (UniqueName: \"kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.936300 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.936266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:17:54.940096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.940081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:54.942471 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.942448 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-967ab780-dockercfg-vvwfw\"" Sep 08 02:17:54.942616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.942487 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-967ab780\"" Sep 08 02:17:54.948666 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.948639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:17:54.953506 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.953481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:54.956921 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:54.956883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") pod \"maas-api-e2e-mt-967ab780-5f46456974-zqjqf\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:55.054378 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.054334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdjf\" (UniqueName: \"kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.054555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.054454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.154935 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.154902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.155071 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.155041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdjf\" (UniqueName: \"kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.155819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.155793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.168398 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.168372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdjf\" (UniqueName: \"kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf\") pod \"payload-processing-e2e-mt-967ab780-c44545b65-czzjh\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.173168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.173143 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:55.251828 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.251802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.445933 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.445905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:17:55.447676 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:55.447650 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfae7f97_54bb_46d9_945f_73651491f0c9.slice/crio-51f628bf5b8df48de7362d65348f8c087725a665347d6bca28a331e3862b556d WatchSource:0}: Error finding container 51f628bf5b8df48de7362d65348f8c087725a665347d6bca28a331e3862b556d: Status 404 returned error can't find the container with id 51f628bf5b8df48de7362d65348f8c087725a665347d6bca28a331e3862b556d Sep 08 02:17:55.534622 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.534598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:17:55.536184 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:17:55.536158 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ec7c49_0d24_4a7c_a2c7_b5781d7ebf80.slice/crio-3f3860684823d39b27c73a9e639908c392716ad83790cbff7ca046824ec3ec18 WatchSource:0}: Error finding container 3f3860684823d39b27c73a9e639908c392716ad83790cbff7ca046824ec3ec18: Status 404 returned error can't find the container with id 3f3860684823d39b27c73a9e639908c392716ad83790cbff7ca046824ec3ec18 Sep 08 02:17:55.988831 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.988799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" event={"ID":"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80","Type":"ContainerStarted","Data":"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b"} Sep 08 02:17:55.989005 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.988859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" event={"ID":"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80","Type":"ContainerStarted","Data":"3f3860684823d39b27c73a9e639908c392716ad83790cbff7ca046824ec3ec18"} Sep 08 02:17:55.989096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.989008 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:17:55.990241 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:55.990215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" event={"ID":"dfae7f97-54bb-46d9-945f-73651491f0c9","Type":"ContainerStarted","Data":"51f628bf5b8df48de7362d65348f8c087725a665347d6bca28a331e3862b556d"} Sep 08 02:17:56.008493 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:56.008438 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" podStartSLOduration=2.008422132 podStartE2EDuration="2.008422132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:56.005051311 +0000 UTC m=+1157.182363385" watchObservedRunningTime="2026-09-08 02:17:56.008422132 +0000 UTC m=+1157.185734184" Sep 08 02:17:56.998180 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:56.998144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" event={"ID":"dfae7f97-54bb-46d9-945f-73651491f0c9","Type":"ContainerStarted","Data":"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c"} Sep 08 02:17:56.998616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:56.998241 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:17:57.016164 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:17:57.016116 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" podStartSLOduration=2.042720401 podStartE2EDuration="3.016103357s" totalImagesPullingTime="973.382956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:57.014470184 +0000 UTC m=+1158.191782270" watchObservedRunningTime="2026-09-08 02:17:57.016103357 +0000 UTC m=+1158.193415429" Sep 08 02:18:03.010470 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:18:03.010391 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:18:07.000177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:18:07.000146 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:18:39.389412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:18:39.389381 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:18:39.390116 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:18:39.390098 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:19:07.512951 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.512911 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:19:07.513484 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.513191 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" podUID="dfae7f97-54bb-46d9-945f-73651491f0c9" containerName="maas-api" containerID="cri-o://410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c" gracePeriod=30 Sep 08 02:19:07.567673 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.567635 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:19:07.567976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.567946 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" podUID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" containerName="payload-processing" containerID="cri-o://9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b" gracePeriod=30 Sep 08 02:19:07.830654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.830626 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:19:07.850132 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.849776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca\") pod \"dfae7f97-54bb-46d9-945f-73651491f0c9\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " Sep 08 02:19:07.850132 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.849867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") pod \"dfae7f97-54bb-46d9-945f-73651491f0c9\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " Sep 08 02:19:07.850132 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.849908 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jthn7\" (UniqueName: \"kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7\") pod \"dfae7f97-54bb-46d9-945f-73651491f0c9\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " Sep 08 02:19:07.850132 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.849968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs\") pod \"dfae7f97-54bb-46d9-945f-73651491f0c9\" (UID: \"dfae7f97-54bb-46d9-945f-73651491f0c9\") " Sep 08 02:19:07.850827 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.850671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfae7f97-54bb-46d9-945f-73651491f0c9" (UID: "dfae7f97-54bb-46d9-945f-73651491f0c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:19:07.851171 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.851142 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfae7f97-54bb-46d9-945f-73651491f0c9-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:07.852936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.852908 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfae7f97-54bb-46d9-945f-73651491f0c9" (UID: "dfae7f97-54bb-46d9-945f-73651491f0c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:19:07.852936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.852920 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7" (OuterVolumeSpecName: "kube-api-access-jthn7") pod "dfae7f97-54bb-46d9-945f-73651491f0c9" (UID: "dfae7f97-54bb-46d9-945f-73651491f0c9"). InnerVolumeSpecName "kube-api-access-jthn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:19:07.854315 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.854267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfae7f97-54bb-46d9-945f-73651491f0c9" (UID: "dfae7f97-54bb-46d9-945f-73651491f0c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:19:07.867674 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.867650 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:19:07.951988 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.951945 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xdjf\" (UniqueName: \"kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf\") pod \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " Sep 08 02:19:07.952190 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.952046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume\") pod \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\" (UID: \"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80\") " Sep 08 02:19:07.952355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.952339 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:07.952432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.952360 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthn7\" (UniqueName: \"kubernetes.io/projected/dfae7f97-54bb-46d9-945f-73651491f0c9-kube-api-access-jthn7\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:07.952432 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.952376 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfae7f97-54bb-46d9-945f-73651491f0c9-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:07.952509 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.952422 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" (UID: "60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:19:07.954002 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:07.953982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf" (OuterVolumeSpecName: "kube-api-access-9xdjf") pod "60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" (UID: "60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80"). InnerVolumeSpecName "kube-api-access-9xdjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:19:08.053359 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.053268 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xdjf\" (UniqueName: \"kubernetes.io/projected/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-kube-api-access-9xdjf\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:08.053359 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.053307 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:19:08.452304 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.452253 2559 generic.go:356] "Generic (PLEG): container finished" podID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" containerID="9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b" exitCode=0 Sep 08 02:19:08.452304 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.452309 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" Sep 08 02:19:08.452602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.452345 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" event={"ID":"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80","Type":"ContainerDied","Data":"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b"} Sep 08 02:19:08.452602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.452408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh" event={"ID":"60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80","Type":"ContainerDied","Data":"3f3860684823d39b27c73a9e639908c392716ad83790cbff7ca046824ec3ec18"} Sep 08 02:19:08.452602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.452436 2559 scope.go:117] "RemoveContainer" containerID="9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b" Sep 08 02:19:08.453729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.453704 2559 generic.go:356] "Generic (PLEG): container finished" podID="dfae7f97-54bb-46d9-945f-73651491f0c9" containerID="410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c" exitCode=0 Sep 08 02:19:08.453860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.453736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" event={"ID":"dfae7f97-54bb-46d9-945f-73651491f0c9","Type":"ContainerDied","Data":"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c"} Sep 08 02:19:08.453860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.453782 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" event={"ID":"dfae7f97-54bb-46d9-945f-73651491f0c9","Type":"ContainerDied","Data":"51f628bf5b8df48de7362d65348f8c087725a665347d6bca28a331e3862b556d"} Sep 08 02:19:08.453860 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.453798 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf" Sep 08 02:19:08.474552 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.474531 2559 scope.go:117] "RemoveContainer" containerID="9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b" Sep 08 02:19:08.474885 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:19:08.474863 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b\": container with ID starting with 9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b not found: ID does not exist" containerID="9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b" Sep 08 02:19:08.474944 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.474892 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b"} err="failed to get container status \"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b\": rpc error: code = NotFound desc = could not find container \"9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b\": container with ID starting with 9ffd27693928157a5affbc483ea5ef0cb66bd61f9c7b15dc1457d231db7ab58b not found: ID does not exist" Sep 08 02:19:08.474944 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.474912 2559 scope.go:117] "RemoveContainer" containerID="410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c" Sep 08 02:19:08.483641 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.483623 2559 scope.go:117] "RemoveContainer" containerID="410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c" Sep 08 02:19:08.483937 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:19:08.483918 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c\": container with ID starting with 410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c not found: ID does not exist" containerID="410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c" Sep 08 02:19:08.483981 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.483945 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c"} err="failed to get container status \"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c\": rpc error: code = NotFound desc = could not find container \"410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c\": container with ID starting with 410870f71f187d73056a5149ff9ad515231e8463b8f21f2ea4cd52999b342f1c not found: ID does not exist" Sep 08 02:19:08.485732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.485705 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:19:08.489048 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.489023 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-967ab780-5f46456974-zqjqf"] Sep 08 02:19:08.501373 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.501347 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:19:08.504823 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:08.504801 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-967ab780-c44545b65-czzjh"] Sep 08 02:19:09.320590 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:09.320553 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" path="/var/lib/kubelet/pods/60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80/volumes" Sep 08 02:19:09.320969 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:19:09.320954 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfae7f97-54bb-46d9-945f-73651491f0c9" path="/var/lib/kubelet/pods/dfae7f97-54bb-46d9-945f-73651491f0c9/volumes" Sep 08 02:20:08.337864 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.337825 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7682-039a9f"] Sep 08 02:20:08.338518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338499 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfae7f97-54bb-46d9-945f-73651491f0c9" containerName="maas-api" Sep 08 02:20:08.338572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338522 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfae7f97-54bb-46d9-945f-73651491f0c9" containerName="maas-api" Sep 08 02:20:08.338572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338548 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" containerName="payload-processing" Sep 08 02:20:08.338572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338558 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" containerName="payload-processing" Sep 08 02:20:08.338700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338687 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ec7c49-0d24-4a7c-a2c7-b5781d7ebf80" containerName="payload-processing" Sep 08 02:20:08.338736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.338704 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfae7f97-54bb-46d9-945f-73651491f0c9" containerName="maas-api" Sep 08 02:20:08.343654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.343632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:08.345917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.345894 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2574n\"" Sep 08 02:20:08.346160 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.346131 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-039a9f"] Sep 08 02:20:08.424106 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.424075 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2vk\" (UniqueName: \"kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk\") pod \"mt-curl-7682-039a9f\" (UID: \"354df363-b1e3-4be5-abde-9d70cca460d1\") " pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:08.524898 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.524860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2vk\" (UniqueName: \"kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk\") pod \"mt-curl-7682-039a9f\" (UID: \"354df363-b1e3-4be5-abde-9d70cca460d1\") " pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:08.533005 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.532981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2vk\" (UniqueName: \"kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk\") pod \"mt-curl-7682-039a9f\" (UID: \"354df363-b1e3-4be5-abde-9d70cca460d1\") " pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:08.656752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.656730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:08.928181 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:08.928158 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-039a9f"] Sep 08 02:20:08.930182 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:20:08.930153 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod354df363_b1e3_4be5_abde_9d70cca460d1.slice/crio-9ee49fa772d1c45fa23b574dd4ee972f09e97e41ab8008e614c30fe76c3e3aba WatchSource:0}: Error finding container 9ee49fa772d1c45fa23b574dd4ee972f09e97e41ab8008e614c30fe76c3e3aba: Status 404 returned error can't find the container with id 9ee49fa772d1c45fa23b574dd4ee972f09e97e41ab8008e614c30fe76c3e3aba Sep 08 02:20:09.822199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:09.822168 2559 generic.go:356] "Generic (PLEG): container finished" podID="354df363-b1e3-4be5-abde-9d70cca460d1" containerID="616506a975b08833133052b2d35d4549e685329cccbc844f1e8e17754f9ad9fb" exitCode=0 Sep 08 02:20:09.822560 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:09.822215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-039a9f" event={"ID":"354df363-b1e3-4be5-abde-9d70cca460d1","Type":"ContainerDied","Data":"616506a975b08833133052b2d35d4549e685329cccbc844f1e8e17754f9ad9fb"} Sep 08 02:20:09.822560 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:09.822252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-039a9f" event={"ID":"354df363-b1e3-4be5-abde-9d70cca460d1","Type":"ContainerStarted","Data":"9ee49fa772d1c45fa23b574dd4ee972f09e97e41ab8008e614c30fe76c3e3aba"} Sep 08 02:20:10.963847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:10.963822 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:10.992159 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:10.992134 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7682-039a9f_354df363-b1e3-4be5-abde-9d70cca460d1/mt-curl-7682-039a9f/0.log" Sep 08 02:20:11.013957 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.013930 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7682-039a9f"] Sep 08 02:20:11.016776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.016754 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7682-039a9f"] Sep 08 02:20:11.049652 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.049630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s2vk\" (UniqueName: \"kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk\") pod \"354df363-b1e3-4be5-abde-9d70cca460d1\" (UID: \"354df363-b1e3-4be5-abde-9d70cca460d1\") " Sep 08 02:20:11.051645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.051612 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk" (OuterVolumeSpecName: "kube-api-access-8s2vk") pod "354df363-b1e3-4be5-abde-9d70cca460d1" (UID: "354df363-b1e3-4be5-abde-9d70cca460d1"). InnerVolumeSpecName "kube-api-access-8s2vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:20:11.150522 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.150481 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s2vk\" (UniqueName: \"kubernetes.io/projected/354df363-b1e3-4be5-abde-9d70cca460d1-kube-api-access-8s2vk\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:20:11.319728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.319700 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354df363-b1e3-4be5-abde-9d70cca460d1" path="/var/lib/kubelet/pods/354df363-b1e3-4be5-abde-9d70cca460d1/volumes" Sep 08 02:20:11.835567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.835541 2559 scope.go:117] "RemoveContainer" containerID="616506a975b08833133052b2d35d4549e685329cccbc844f1e8e17754f9ad9fb" Sep 08 02:20:11.835567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:11.835556 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-039a9f" Sep 08 02:20:55.321554 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.321519 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:20:55.322179 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.322162 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354df363-b1e3-4be5-abde-9d70cca460d1" containerName="mt-curl-7682-039a9f" Sep 08 02:20:55.322225 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.322183 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354df363-b1e3-4be5-abde-9d70cca460d1" containerName="mt-curl-7682-039a9f" Sep 08 02:20:55.322330 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.322285 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="354df363-b1e3-4be5-abde-9d70cca460d1" containerName="mt-curl-7682-039a9f" Sep 08 02:20:55.327752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.327725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.329950 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.329927 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c25e6e-serving-cert\"" Sep 08 02:20:55.333777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.333748 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:20:55.363828 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.363794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.364006 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.363874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.364070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.364045 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgmm\" (UniqueName: \"kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.364222 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.364199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.464809 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.464777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.465002 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.464824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.465002 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.464869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.465002 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.464901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgmm\" (UniqueName: \"kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.465190 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:20:55.465014 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-serving-cert: secret "maas-api-e2e-api-a-c25e6e-serving-cert" not found Sep 08 02:20:55.465190 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:20:55.465092 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls podName:bfb86ba0-cb9d-40c3-a2c2-572ea7a75659 nodeName:}" failed. No retries permitted until 2026-09-08 02:20:55.965071725 +0000 UTC m=+1337.142383778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls") pod "maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" (UID: "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659") : secret "maas-api-e2e-api-a-c25e6e-serving-cert" not found Sep 08 02:20:55.465607 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.465582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.467305 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.467272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.472890 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.472864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgmm\" (UniqueName: \"kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.939282 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.939215 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:20:55.943463 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.943445 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:55.945663 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.945646 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c25e6e-dockercfg-xjn54\"" Sep 08 02:20:55.945762 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.945682 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c25e6e\"" Sep 08 02:20:55.951649 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.951418 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:20:55.970522 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.970488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:55.973377 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:55.973351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") pod \"maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:56.071601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.071564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmdf\" (UniqueName: \"kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.071776 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.071629 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.173193 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.173156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.173409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.173284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmdf\" (UniqueName: \"kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.173936 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.173914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.182891 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.182863 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmdf\" (UniqueName: \"kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf\") pod \"payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.241893 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.241797 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:56.256266 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.256241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:56.529075 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.529047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:20:56.529706 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:20:56.529683 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb86ba0_cb9d_40c3_a2c2_572ea7a75659.slice/crio-fcf7750d57adc8cae9425ef76d5d860001733ee11de9150255bdd0c8a1cab06e WatchSource:0}: Error finding container fcf7750d57adc8cae9425ef76d5d860001733ee11de9150255bdd0c8a1cab06e: Status 404 returned error can't find the container with id fcf7750d57adc8cae9425ef76d5d860001733ee11de9150255bdd0c8a1cab06e Sep 08 02:20:56.553029 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:56.553002 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:20:56.554977 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:20:56.554947 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a63279_7e3e_4b16_9ed0_cab1e21700a0.slice/crio-16503e1aeacbc50f4e962550d1b5929e69289422cd7b269d0e4b57fbac1038bf WatchSource:0}: Error finding container 16503e1aeacbc50f4e962550d1b5929e69289422cd7b269d0e4b57fbac1038bf: Status 404 returned error can't find the container with id 16503e1aeacbc50f4e962550d1b5929e69289422cd7b269d0e4b57fbac1038bf Sep 08 02:20:57.125469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.125432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" event={"ID":"09a63279-7e3e-4b16-9ed0-cab1e21700a0","Type":"ContainerStarted","Data":"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff"} Sep 08 02:20:57.125665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.125600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:20:57.125665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.125650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" event={"ID":"09a63279-7e3e-4b16-9ed0-cab1e21700a0","Type":"ContainerStarted","Data":"16503e1aeacbc50f4e962550d1b5929e69289422cd7b269d0e4b57fbac1038bf"} Sep 08 02:20:57.126986 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.126962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" event={"ID":"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659","Type":"ContainerStarted","Data":"2760115bf6f875b68dfe6d8e01fd5cbb600cf8bd7e64d87f282a86d47c9941f7"} Sep 08 02:20:57.127072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.127013 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:20:57.127072 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.127037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" event={"ID":"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659","Type":"ContainerStarted","Data":"fcf7750d57adc8cae9425ef76d5d860001733ee11de9150255bdd0c8a1cab06e"} Sep 08 02:20:57.146483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.146424 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" podStartSLOduration=2.146407996 podStartE2EDuration="2.146407996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:20:57.142073402 +0000 UTC m=+1338.319385475" watchObservedRunningTime="2026-09-08 02:20:57.146407996 +0000 UTC m=+1338.323720068" Sep 08 02:20:57.160630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:20:57.160567 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" podStartSLOduration=1.778689422 podStartE2EDuration="2.160546335s" totalImagesPullingTime="381.856913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:20:57.157222339 +0000 UTC m=+1338.334534433" watchObservedRunningTime="2026-09-08 02:20:57.160546335 +0000 UTC m=+1338.337858407" Sep 08 02:21:03.139390 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:03.139311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:21:08.137070 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:08.137035 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:21:24.538039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.537997 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:21:24.541931 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.541908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.544234 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.544211 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4a198d-serving-cert\"" Sep 08 02:21:24.551578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.551559 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:21:24.640806 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.640775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.640985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.640822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.640985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.640854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gjc\" (UniqueName: \"kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.640985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.640936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.741425 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.741394 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.741612 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.741456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.741612 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.741494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.741612 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.741531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gjc\" (UniqueName: \"kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.742227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.742204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.743887 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.743862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.743974 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.743912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.749165 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.749140 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gjc\" (UniqueName: \"kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc\") pod \"maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:24.856125 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:24.856061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:25.130197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.130168 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:21:25.131826 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:21:25.131800 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9669a6_0f72_44f6_8c32_84eaf03a2e11.slice/crio-0b7bb7b2501aaa480dedec33518b7da802fbde0b764d0b143382e4f970075b4c WatchSource:0}: Error finding container 0b7bb7b2501aaa480dedec33518b7da802fbde0b764d0b143382e4f970075b4c: Status 404 returned error can't find the container with id 0b7bb7b2501aaa480dedec33518b7da802fbde0b764d0b143382e4f970075b4c Sep 08 02:21:25.139941 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.139918 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:21:25.144665 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.144646 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.146784 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.146766 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4a198d\"" Sep 08 02:21:25.153908 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.153890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:21:25.193367 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.193337 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:21:25.196914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.196893 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.199136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.199119 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4a198d-dockercfg-4p8c4\"" Sep 08 02:21:25.207156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.207134 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:21:25.245029 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.244999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9bq\" (UniqueName: \"kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.245173 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.245040 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f2d\" (UniqueName: \"kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.245173 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.245099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.245173 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.245155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.308399 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.308362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" event={"ID":"0b9669a6-0f72-44f6-8c32-84eaf03a2e11","Type":"ContainerStarted","Data":"0b7bb7b2501aaa480dedec33518b7da802fbde0b764d0b143382e4f970075b4c"} Sep 08 02:21:25.346473 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.346439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f2d\" (UniqueName: \"kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.346624 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.346492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.346624 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.346548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.346624 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.346618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9bq\" (UniqueName: \"kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.347265 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.347246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.347351 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.347277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.353999 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.353978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9bq\" (UniqueName: \"kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq\") pod \"payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.354104 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.354012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f2d\" (UniqueName: \"kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d\") pod \"payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.457033 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.457010 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:25.509322 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.509267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:25.736249 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.736224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:21:25.737181 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:21:25.737155 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e2ab18_d48d_4255_a04e_f74344614ed4.slice/crio-9ab66615abbb412676d66db80fa93e169a872538e09e35be8735b1f342c5c033 WatchSource:0}: Error finding container 9ab66615abbb412676d66db80fa93e169a872538e09e35be8735b1f342c5c033: Status 404 returned error can't find the container with id 9ab66615abbb412676d66db80fa93e169a872538e09e35be8735b1f342c5c033 Sep 08 02:21:25.795732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:25.795707 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:21:25.804508 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:21:25.804470 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433fe496_2675_4dcf_9d52_22d661a50b1e.slice/crio-7eee2159e5ed12aa6967f19ca080967a812dc7a89c04a6d6afe286f150066482 WatchSource:0}: Error finding container 7eee2159e5ed12aa6967f19ca080967a812dc7a89c04a6d6afe286f150066482: Status 404 returned error can't find the container with id 7eee2159e5ed12aa6967f19ca080967a812dc7a89c04a6d6afe286f150066482 Sep 08 02:21:26.317579 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.317544 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" event={"ID":"c4e2ab18-d48d-4255-a04e-f74344614ed4","Type":"ContainerStarted","Data":"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd"} Sep 08 02:21:26.317786 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.317600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:26.317786 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.317622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" event={"ID":"c4e2ab18-d48d-4255-a04e-f74344614ed4","Type":"ContainerStarted","Data":"9ab66615abbb412676d66db80fa93e169a872538e09e35be8735b1f342c5c033"} Sep 08 02:21:26.319160 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.319131 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" event={"ID":"0b9669a6-0f72-44f6-8c32-84eaf03a2e11","Type":"ContainerStarted","Data":"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba"} Sep 08 02:21:26.319312 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.319192 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:26.320777 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.320751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" event={"ID":"433fe496-2675-4dcf-9d52-22d661a50b1e","Type":"ContainerStarted","Data":"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124"} Sep 08 02:21:26.320889 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.320802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" event={"ID":"433fe496-2675-4dcf-9d52-22d661a50b1e","Type":"ContainerStarted","Data":"7eee2159e5ed12aa6967f19ca080967a812dc7a89c04a6d6afe286f150066482"} Sep 08 02:21:26.320889 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.320831 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:26.335547 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.335511 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" podStartSLOduration=1.335498676 podStartE2EDuration="1.335498676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:21:26.332354508 +0000 UTC m=+1367.509666581" watchObservedRunningTime="2026-09-08 02:21:26.335498676 +0000 UTC m=+1367.512810748" Sep 08 02:21:26.347227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.347187 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" podStartSLOduration=2.057693674 podStartE2EDuration="2.347173863s" totalImagesPullingTime="289.480189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:21:26.34624829 +0000 UTC m=+1367.523560362" watchObservedRunningTime="2026-09-08 02:21:26.347173863 +0000 UTC m=+1367.524485955" Sep 08 02:21:26.362807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:26.362761 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" podStartSLOduration=1.362744734 podStartE2EDuration="1.362744734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:21:26.360576822 +0000 UTC m=+1367.537888894" watchObservedRunningTime="2026-09-08 02:21:26.362744734 +0000 UTC m=+1367.540056807" Sep 08 02:21:32.091501 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.091464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7682-9fb744"] Sep 08 02:21:32.096408 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.096385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:32.098657 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.098636 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2574n\"" Sep 08 02:21:32.100503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.100477 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-9fb744"] Sep 08 02:21:32.104585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.104522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqgc\" (UniqueName: \"kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc\") pod \"mt-curl-7682-9fb744\" (UID: \"7466d79f-c2b7-4828-a546-d7e8c82fe2db\") " pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:32.205819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.205786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqgc\" (UniqueName: \"kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc\") pod \"mt-curl-7682-9fb744\" (UID: \"7466d79f-c2b7-4828-a546-d7e8c82fe2db\") " pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:32.216849 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.216820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqgc\" (UniqueName: \"kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc\") pod \"mt-curl-7682-9fb744\" (UID: \"7466d79f-c2b7-4828-a546-d7e8c82fe2db\") " pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:32.333467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.333442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:21:32.408028 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.408000 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:32.680541 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.680514 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-9fb744"] Sep 08 02:21:32.682753 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:21:32.682727 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7466d79f_c2b7_4828_a546_d7e8c82fe2db.slice/crio-db47e439247a0ce5a3011b601570c34871f029e524623dad806722bc81c523bd WatchSource:0}: Error finding container db47e439247a0ce5a3011b601570c34871f029e524623dad806722bc81c523bd: Status 404 returned error can't find the container with id db47e439247a0ce5a3011b601570c34871f029e524623dad806722bc81c523bd Sep 08 02:21:32.684205 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:32.684188 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 02:21:33.371216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:33.371119 2559 generic.go:356] "Generic (PLEG): container finished" podID="7466d79f-c2b7-4828-a546-d7e8c82fe2db" containerID="ec62fbbf3eecaf076afd4c62246276d21572574bd0216a916a9e7c1006c70648" exitCode=0 Sep 08 02:21:33.371216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:33.371195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-9fb744" event={"ID":"7466d79f-c2b7-4828-a546-d7e8c82fe2db","Type":"ContainerDied","Data":"ec62fbbf3eecaf076afd4c62246276d21572574bd0216a916a9e7c1006c70648"} Sep 08 02:21:33.371701 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:33.371232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-9fb744" event={"ID":"7466d79f-c2b7-4828-a546-d7e8c82fe2db","Type":"ContainerStarted","Data":"db47e439247a0ce5a3011b601570c34871f029e524623dad806722bc81c523bd"} Sep 08 02:21:34.515828 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.515807 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:34.535234 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.535208 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7682-9fb744_7466d79f-c2b7-4828-a546-d7e8c82fe2db/mt-curl-7682-9fb744/0.log" Sep 08 02:21:34.558189 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.558144 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7682-9fb744"] Sep 08 02:21:34.561966 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.561945 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7682-9fb744"] Sep 08 02:21:34.629000 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.628975 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpqgc\" (UniqueName: \"kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc\") pod \"7466d79f-c2b7-4828-a546-d7e8c82fe2db\" (UID: \"7466d79f-c2b7-4828-a546-d7e8c82fe2db\") " Sep 08 02:21:34.630983 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.630923 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc" (OuterVolumeSpecName: "kube-api-access-gpqgc") pod "7466d79f-c2b7-4828-a546-d7e8c82fe2db" (UID: "7466d79f-c2b7-4828-a546-d7e8c82fe2db"). InnerVolumeSpecName "kube-api-access-gpqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:21:34.677875 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.677854 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7682-30b56d"] Sep 08 02:21:34.678317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.678305 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7466d79f-c2b7-4828-a546-d7e8c82fe2db" containerName="mt-curl-7682-9fb744" Sep 08 02:21:34.678365 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.678319 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7466d79f-c2b7-4828-a546-d7e8c82fe2db" containerName="mt-curl-7682-9fb744" Sep 08 02:21:34.678419 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.678410 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7466d79f-c2b7-4828-a546-d7e8c82fe2db" containerName="mt-curl-7682-9fb744" Sep 08 02:21:34.681698 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.681522 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:34.685333 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.685308 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-30b56d"] Sep 08 02:21:34.729754 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.729725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh8c\" (UniqueName: \"kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c\") pod \"mt-curl-7682-30b56d\" (UID: \"b366c1a0-8bfd-4659-bb79-6eade8d87e08\") " pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:34.729877 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.729779 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqgc\" (UniqueName: \"kubernetes.io/projected/7466d79f-c2b7-4828-a546-d7e8c82fe2db-kube-api-access-gpqgc\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:21:34.830802 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.830772 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh8c\" (UniqueName: \"kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c\") pod \"mt-curl-7682-30b56d\" (UID: \"b366c1a0-8bfd-4659-bb79-6eade8d87e08\") " pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:34.838626 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.838584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh8c\" (UniqueName: \"kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c\") pod \"mt-curl-7682-30b56d\" (UID: \"b366c1a0-8bfd-4659-bb79-6eade8d87e08\") " pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:34.992707 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:34.992678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:35.263993 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:35.263968 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7682-30b56d"] Sep 08 02:21:35.265334 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:21:35.265285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb366c1a0_8bfd_4659_bb79_6eade8d87e08.slice/crio-179fd505b456eef2f705bd8f2043eade4d132d0af02499897c631467db5b5e84 WatchSource:0}: Error finding container 179fd505b456eef2f705bd8f2043eade4d132d0af02499897c631467db5b5e84: Status 404 returned error can't find the container with id 179fd505b456eef2f705bd8f2043eade4d132d0af02499897c631467db5b5e84 Sep 08 02:21:35.320895 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:35.320865 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7466d79f-c2b7-4828-a546-d7e8c82fe2db" path="/var/lib/kubelet/pods/7466d79f-c2b7-4828-a546-d7e8c82fe2db/volumes" Sep 08 02:21:35.386028 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:35.386009 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-9fb744" Sep 08 02:21:35.386146 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:35.386018 2559 scope.go:117] "RemoveContainer" containerID="ec62fbbf3eecaf076afd4c62246276d21572574bd0216a916a9e7c1006c70648" Sep 08 02:21:35.387346 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:35.387317 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-30b56d" event={"ID":"b366c1a0-8bfd-4659-bb79-6eade8d87e08","Type":"ContainerStarted","Data":"179fd505b456eef2f705bd8f2043eade4d132d0af02499897c631467db5b5e84"} Sep 08 02:21:36.395209 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:36.395173 2559 generic.go:356] "Generic (PLEG): container finished" podID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" containerID="42bc2829ad29ab491739d16641dbcc7f39333608cb527b39274ee55f23220d20" exitCode=0 Sep 08 02:21:36.395608 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:36.395233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7682-30b56d" event={"ID":"b366c1a0-8bfd-4659-bb79-6eade8d87e08","Type":"ContainerDied","Data":"42bc2829ad29ab491739d16641dbcc7f39333608cb527b39274ee55f23220d20"} Sep 08 02:21:37.329851 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.329825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:21:37.330027 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.329891 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:21:37.550430 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.550406 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:37.568196 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.568142 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7682-30b56d_b366c1a0-8bfd-4659-bb79-6eade8d87e08/mt-curl-7682-30b56d/0.log" Sep 08 02:21:37.591063 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.590977 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7682-30b56d"] Sep 08 02:21:37.593720 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.593698 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7682-30b56d"] Sep 08 02:21:37.660015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.659980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hh8c\" (UniqueName: \"kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c\") pod \"b366c1a0-8bfd-4659-bb79-6eade8d87e08\" (UID: \"b366c1a0-8bfd-4659-bb79-6eade8d87e08\") " Sep 08 02:21:37.662084 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.662050 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c" (OuterVolumeSpecName: "kube-api-access-9hh8c") pod "b366c1a0-8bfd-4659-bb79-6eade8d87e08" (UID: "b366c1a0-8bfd-4659-bb79-6eade8d87e08"). InnerVolumeSpecName "kube-api-access-9hh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:21:37.761111 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:37.761070 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh8c\" (UniqueName: \"kubernetes.io/projected/b366c1a0-8bfd-4659-bb79-6eade8d87e08-kube-api-access-9hh8c\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:21:38.411729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:38.411701 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:38.411901 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:38.411754 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="179fd505b456eef2f705bd8f2043eade4d132d0af02499897c631467db5b5e84" Sep 08 02:21:38.421556 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:21:38.421528 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7682-30b56d\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:39.318658 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:21:39.318621 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7682-30b56d\" is forbidden: User \"system:node:ip-10-0-132-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-162.ec2.internal' and this object" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" pod="opendatahub/mt-curl-7682-30b56d" Sep 08 02:21:39.320687 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:21:39.320660 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" path="/var/lib/kubelet/pods/b366c1a0-8bfd-4659-bb79-6eade8d87e08/volumes" Sep 08 02:22:06.438179 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.438142 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:22:06.442096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.438437 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" podUID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" containerName="maas-api" containerID="cri-o://2760115bf6f875b68dfe6d8e01fd5cbb600cf8bd7e64d87f282a86d47c9941f7" gracePeriod=30 Sep 08 02:22:06.496474 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.495959 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:22:06.496474 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.496204 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" podUID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" containerName="payload-pre-processing" containerID="cri-o://cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff" gracePeriod=30 Sep 08 02:22:06.602648 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.602614 2559 generic.go:356] "Generic (PLEG): container finished" podID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" containerID="2760115bf6f875b68dfe6d8e01fd5cbb600cf8bd7e64d87f282a86d47c9941f7" exitCode=0 Sep 08 02:22:06.602810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.602690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" event={"ID":"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659","Type":"ContainerDied","Data":"2760115bf6f875b68dfe6d8e01fd5cbb600cf8bd7e64d87f282a86d47c9941f7"} Sep 08 02:22:06.757622 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.757591 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:22:06.799573 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.799551 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:22:06.948543 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948503 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs\") pod \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " Sep 08 02:22:06.948543 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948542 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca\") pod \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " Sep 08 02:22:06.948850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") pod \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " Sep 08 02:22:06.948850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948696 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tgmm\" (UniqueName: \"kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm\") pod \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\" (UID: \"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659\") " Sep 08 02:22:06.948850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948721 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fmdf\" (UniqueName: \"kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf\") pod \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " Sep 08 02:22:06.948850 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume\") pod \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\" (UID: \"09a63279-7e3e-4b16-9ed0-cab1e21700a0\") " Sep 08 02:22:06.949071 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.948925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" (UID: "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:06.949229 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.949196 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:06.949387 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.949258 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09a63279-7e3e-4b16-9ed0-cab1e21700a0" (UID: "09a63279-7e3e-4b16-9ed0-cab1e21700a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:06.950941 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.950913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" (UID: "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:06.951042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.950932 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" (UID: "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:06.951042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.950941 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf" (OuterVolumeSpecName: "kube-api-access-6fmdf") pod "09a63279-7e3e-4b16-9ed0-cab1e21700a0" (UID: "09a63279-7e3e-4b16-9ed0-cab1e21700a0"). InnerVolumeSpecName "kube-api-access-6fmdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:06.951042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:06.950957 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm" (OuterVolumeSpecName: "kube-api-access-6tgmm") pod "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" (UID: "bfb86ba0-cb9d-40c3-a2c2-572ea7a75659"). InnerVolumeSpecName "kube-api-access-6tgmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:07.049963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.049885 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:07.049963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.049924 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:07.049963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.049939 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgmm\" (UniqueName: \"kubernetes.io/projected/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659-kube-api-access-6tgmm\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:07.049963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.049955 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmdf\" (UniqueName: \"kubernetes.io/projected/09a63279-7e3e-4b16-9ed0-cab1e21700a0-kube-api-access-6fmdf\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:07.050207 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.049970 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a63279-7e3e-4b16-9ed0-cab1e21700a0-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:07.611164 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.611075 2559 generic.go:356] "Generic (PLEG): container finished" podID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" containerID="cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff" exitCode=0 Sep 08 02:22:07.611164 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.611133 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" Sep 08 02:22:07.611692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.611169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" event={"ID":"09a63279-7e3e-4b16-9ed0-cab1e21700a0","Type":"ContainerDied","Data":"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff"} Sep 08 02:22:07.611692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.611262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp" event={"ID":"09a63279-7e3e-4b16-9ed0-cab1e21700a0","Type":"ContainerDied","Data":"16503e1aeacbc50f4e962550d1b5929e69289422cd7b269d0e4b57fbac1038bf"} Sep 08 02:22:07.611692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.611322 2559 scope.go:117] "RemoveContainer" containerID="cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff" Sep 08 02:22:07.612834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.612817 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" Sep 08 02:22:07.612898 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.612851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l" event={"ID":"bfb86ba0-cb9d-40c3-a2c2-572ea7a75659","Type":"ContainerDied","Data":"fcf7750d57adc8cae9425ef76d5d860001733ee11de9150255bdd0c8a1cab06e"} Sep 08 02:22:07.621087 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.621071 2559 scope.go:117] "RemoveContainer" containerID="cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff" Sep 08 02:22:07.621389 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:22:07.621372 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff\": container with ID starting with cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff not found: ID does not exist" containerID="cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff" Sep 08 02:22:07.621468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.621400 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff"} err="failed to get container status \"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff\": rpc error: code = NotFound desc = could not find container \"cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff\": container with ID starting with cb8ec6ce85f190445ffaa1c9da0223665305101b966422a867fd8408de9d61ff not found: ID does not exist" Sep 08 02:22:07.621468 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.621415 2559 scope.go:117] "RemoveContainer" containerID="2760115bf6f875b68dfe6d8e01fd5cbb600cf8bd7e64d87f282a86d47c9941f7" Sep 08 02:22:07.633419 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.633393 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:22:07.639815 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.639794 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c25e6e-665958bd7-x9p2l"] Sep 08 02:22:07.649367 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.649336 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:22:07.652685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:07.652666 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c25e6e-59454fdf54-x2pzp"] Sep 08 02:22:09.320124 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:09.320088 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" path="/var/lib/kubelet/pods/09a63279-7e3e-4b16-9ed0-cab1e21700a0/volumes" Sep 08 02:22:09.320572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:09.320555 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" path="/var/lib/kubelet/pods/bfb86ba0-cb9d-40c3-a2c2-572ea7a75659/volumes" Sep 08 02:22:28.968592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:28.968543 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:22:28.969034 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:28.968862 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" podUID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" containerName="maas-api" containerID="cri-o://e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba" gracePeriod=30 Sep 08 02:22:29.025208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.025176 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:22:29.025481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.025450 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" podUID="433fe496-2675-4dcf-9d52-22d661a50b1e" containerName="payload-processing" containerID="cri-o://f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124" gracePeriod=30 Sep 08 02:22:29.030845 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.030815 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:22:29.031107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.031033 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" podUID="c4e2ab18-d48d-4255-a04e-f74344614ed4" containerName="payload-pre-processing" containerID="cri-o://2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd" gracePeriod=30 Sep 08 02:22:29.318095 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.318065 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:22:29.333158 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.333134 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:22:29.340245 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.340218 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:22:29.470872 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.470843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca\") pod \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " Sep 08 02:22:29.471065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.470883 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5f2d\" (UniqueName: \"kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d\") pod \"c4e2ab18-d48d-4255-a04e-f74344614ed4\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " Sep 08 02:22:29.471065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.470918 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9bq\" (UniqueName: \"kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq\") pod \"433fe496-2675-4dcf-9d52-22d661a50b1e\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " Sep 08 02:22:29.471065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.470954 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls\") pod \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " Sep 08 02:22:29.471065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.470992 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs\") pod \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " Sep 08 02:22:29.471065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.471020 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume\") pod \"c4e2ab18-d48d-4255-a04e-f74344614ed4\" (UID: \"c4e2ab18-d48d-4255-a04e-f74344614ed4\") " Sep 08 02:22:29.471342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.471143 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8gjc\" (UniqueName: \"kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc\") pod \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\" (UID: \"0b9669a6-0f72-44f6-8c32-84eaf03a2e11\") " Sep 08 02:22:29.471342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.471189 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume\") pod \"433fe496-2675-4dcf-9d52-22d661a50b1e\" (UID: \"433fe496-2675-4dcf-9d52-22d661a50b1e\") " Sep 08 02:22:29.471342 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.471276 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b9669a6-0f72-44f6-8c32-84eaf03a2e11" (UID: "0b9669a6-0f72-44f6-8c32-84eaf03a2e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:29.471521 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.471489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4e2ab18-d48d-4255-a04e-f74344614ed4" (UID: "c4e2ab18-d48d-4255-a04e-f74344614ed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:29.472536 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.472509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "433fe496-2675-4dcf-9d52-22d661a50b1e" (UID: "433fe496-2675-4dcf-9d52-22d661a50b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:29.472663 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.472637 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e2ab18-d48d-4255-a04e-f74344614ed4-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.472663 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.472658 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433fe496-2675-4dcf-9d52-22d661a50b1e-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.472779 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.472676 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.473561 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.473522 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0b9669a6-0f72-44f6-8c32-84eaf03a2e11" (UID: "0b9669a6-0f72-44f6-8c32-84eaf03a2e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:29.473674 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.473653 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d" (OuterVolumeSpecName: "kube-api-access-w5f2d") pod "c4e2ab18-d48d-4255-a04e-f74344614ed4" (UID: "c4e2ab18-d48d-4255-a04e-f74344614ed4"). InnerVolumeSpecName "kube-api-access-w5f2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:29.473781 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.473761 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc" (OuterVolumeSpecName: "kube-api-access-j8gjc") pod "0b9669a6-0f72-44f6-8c32-84eaf03a2e11" (UID: "0b9669a6-0f72-44f6-8c32-84eaf03a2e11"). InnerVolumeSpecName "kube-api-access-j8gjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:29.474210 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.474186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b9669a6-0f72-44f6-8c32-84eaf03a2e11" (UID: "0b9669a6-0f72-44f6-8c32-84eaf03a2e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:29.474622 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.474602 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq" (OuterVolumeSpecName: "kube-api-access-2h9bq") pod "433fe496-2675-4dcf-9d52-22d661a50b1e" (UID: "433fe496-2675-4dcf-9d52-22d661a50b1e"). InnerVolumeSpecName "kube-api-access-2h9bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:29.573487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.573389 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9bq\" (UniqueName: \"kubernetes.io/projected/433fe496-2675-4dcf-9d52-22d661a50b1e-kube-api-access-2h9bq\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.573487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.573431 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.573487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.573446 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.573487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.573459 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gjc\" (UniqueName: \"kubernetes.io/projected/0b9669a6-0f72-44f6-8c32-84eaf03a2e11-kube-api-access-j8gjc\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.573487 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.573472 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f2d\" (UniqueName: \"kubernetes.io/projected/c4e2ab18-d48d-4255-a04e-f74344614ed4-kube-api-access-w5f2d\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:29.761146 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.761114 2559 generic.go:356] "Generic (PLEG): container finished" podID="c4e2ab18-d48d-4255-a04e-f74344614ed4" containerID="2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd" exitCode=0 Sep 08 02:22:29.761362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.761177 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" Sep 08 02:22:29.761362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.761209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" event={"ID":"c4e2ab18-d48d-4255-a04e-f74344614ed4","Type":"ContainerDied","Data":"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd"} Sep 08 02:22:29.761362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.761267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n" event={"ID":"c4e2ab18-d48d-4255-a04e-f74344614ed4","Type":"ContainerDied","Data":"9ab66615abbb412676d66db80fa93e169a872538e09e35be8735b1f342c5c033"} Sep 08 02:22:29.761362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.761318 2559 scope.go:117] "RemoveContainer" containerID="2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd" Sep 08 02:22:29.762847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.762822 2559 generic.go:356] "Generic (PLEG): container finished" podID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" containerID="e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba" exitCode=0 Sep 08 02:22:29.762973 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.762861 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" Sep 08 02:22:29.762973 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.762907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" event={"ID":"0b9669a6-0f72-44f6-8c32-84eaf03a2e11","Type":"ContainerDied","Data":"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba"} Sep 08 02:22:29.762973 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.762963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf" event={"ID":"0b9669a6-0f72-44f6-8c32-84eaf03a2e11","Type":"ContainerDied","Data":"0b7bb7b2501aaa480dedec33518b7da802fbde0b764d0b143382e4f970075b4c"} Sep 08 02:22:29.765101 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.765069 2559 generic.go:356] "Generic (PLEG): container finished" podID="433fe496-2675-4dcf-9d52-22d661a50b1e" containerID="f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124" exitCode=0 Sep 08 02:22:29.765199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.765183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" event={"ID":"433fe496-2675-4dcf-9d52-22d661a50b1e","Type":"ContainerDied","Data":"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124"} Sep 08 02:22:29.765273 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.765228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" event={"ID":"433fe496-2675-4dcf-9d52-22d661a50b1e","Type":"ContainerDied","Data":"7eee2159e5ed12aa6967f19ca080967a812dc7a89c04a6d6afe286f150066482"} Sep 08 02:22:29.765273 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.765195 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k" Sep 08 02:22:29.773362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.773343 2559 scope.go:117] "RemoveContainer" containerID="2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd" Sep 08 02:22:29.773677 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:22:29.773650 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd\": container with ID starting with 2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd not found: ID does not exist" containerID="2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd" Sep 08 02:22:29.773769 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.773688 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd"} err="failed to get container status \"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd\": rpc error: code = NotFound desc = could not find container \"2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd\": container with ID starting with 2eb87c325c0058c66e0cd91752b952069e1720b0bcb97163c452a16dbd1774dd not found: ID does not exist" Sep 08 02:22:29.773769 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.773710 2559 scope.go:117] "RemoveContainer" containerID="e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba" Sep 08 02:22:29.783454 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.783431 2559 scope.go:117] "RemoveContainer" containerID="e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba" Sep 08 02:22:29.783766 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:22:29.783745 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba\": container with ID starting with e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba not found: ID does not exist" containerID="e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba" Sep 08 02:22:29.783834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.783778 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba"} err="failed to get container status \"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba\": rpc error: code = NotFound desc = could not find container \"e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba\": container with ID starting with e4b26e2446267d129689913eae42efb9d2b9ad1aa3cc4d87abfdd391e57d29ba not found: ID does not exist" Sep 08 02:22:29.783834 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.783804 2559 scope.go:117] "RemoveContainer" containerID="f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124" Sep 08 02:22:29.785880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.785857 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:22:29.791480 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.791453 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a198d-6cb547947d-7wq9n"] Sep 08 02:22:29.792700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.792685 2559 scope.go:117] "RemoveContainer" containerID="f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124" Sep 08 02:22:29.792984 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:22:29.792968 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124\": container with ID starting with f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124 not found: ID does not exist" containerID="f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124" Sep 08 02:22:29.793030 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.792990 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124"} err="failed to get container status \"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124\": rpc error: code = NotFound desc = could not find container \"f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124\": container with ID starting with f7f6a5a5d4e6d0dfb9e19d6a92236ca2d5e6b2d2b187685795030c87424e8124 not found: ID does not exist" Sep 08 02:22:29.800306 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.800269 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:22:29.804537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.804515 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4a198d-6f8fbfd47-6xl7k"] Sep 08 02:22:29.814537 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.814518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:22:29.818095 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:29.818075 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a198d-58fccfdb4c-t6jlf"] Sep 08 02:22:31.320460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:31.320427 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" path="/var/lib/kubelet/pods/0b9669a6-0f72-44f6-8c32-84eaf03a2e11/volumes" Sep 08 02:22:31.320833 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:31.320786 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433fe496-2675-4dcf-9d52-22d661a50b1e" path="/var/lib/kubelet/pods/433fe496-2675-4dcf-9d52-22d661a50b1e/volumes" Sep 08 02:22:31.321133 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:31.321119 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e2ab18-d48d-4255-a04e-f74344614ed4" path="/var/lib/kubelet/pods/c4e2ab18-d48d-4255-a04e-f74344614ed4/volumes" Sep 08 02:22:46.228567 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.228528 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:22:46.229473 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.229417 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" podUID="60710743-598c-4821-aa9a-67917db6fd52" containerName="maas-api" containerID="cri-o://9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa" gracePeriod=30 Sep 08 02:22:46.495813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.495785 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:22:46.534138 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.534101 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") pod \"60710743-598c-4821-aa9a-67917db6fd52\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " Sep 08 02:22:46.534368 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.534162 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca\") pod \"60710743-598c-4821-aa9a-67917db6fd52\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " Sep 08 02:22:46.534368 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.534192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs\") pod \"60710743-598c-4821-aa9a-67917db6fd52\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " Sep 08 02:22:46.534368 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.534231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvgr4\" (UniqueName: \"kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4\") pod \"60710743-598c-4821-aa9a-67917db6fd52\" (UID: \"60710743-598c-4821-aa9a-67917db6fd52\") " Sep 08 02:22:46.535016 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.534980 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60710743-598c-4821-aa9a-67917db6fd52" (UID: "60710743-598c-4821-aa9a-67917db6fd52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:22:46.536453 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.536431 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60710743-598c-4821-aa9a-67917db6fd52" (UID: "60710743-598c-4821-aa9a-67917db6fd52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:46.536572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.536543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4" (OuterVolumeSpecName: "kube-api-access-lvgr4") pod "60710743-598c-4821-aa9a-67917db6fd52" (UID: "60710743-598c-4821-aa9a-67917db6fd52"). InnerVolumeSpecName "kube-api-access-lvgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:22:46.537088 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.537071 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60710743-598c-4821-aa9a-67917db6fd52" (UID: "60710743-598c-4821-aa9a-67917db6fd52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:22:46.635029 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.634994 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60710743-598c-4821-aa9a-67917db6fd52-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:46.635029 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.635025 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:46.635237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.635038 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgr4\" (UniqueName: \"kubernetes.io/projected/60710743-598c-4821-aa9a-67917db6fd52-kube-api-access-lvgr4\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:46.635237 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.635051 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60710743-598c-4821-aa9a-67917db6fd52-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:22:46.868790 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.868712 2559 generic.go:356] "Generic (PLEG): container finished" podID="60710743-598c-4821-aa9a-67917db6fd52" containerID="9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa" exitCode=0 Sep 08 02:22:46.868790 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.868758 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" Sep 08 02:22:46.868790 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.868781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" event={"ID":"60710743-598c-4821-aa9a-67917db6fd52","Type":"ContainerDied","Data":"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa"} Sep 08 02:22:46.869022 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.868824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4" event={"ID":"60710743-598c-4821-aa9a-67917db6fd52","Type":"ContainerDied","Data":"589a4384688eddfdf8ec7b8180c18158804ea6bbb9de249d9c0fd1365cef0716"} Sep 08 02:22:46.869022 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.868846 2559 scope.go:117] "RemoveContainer" containerID="9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa" Sep 08 02:22:46.878677 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.878660 2559 scope.go:117] "RemoveContainer" containerID="9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa" Sep 08 02:22:46.878933 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:22:46.878914 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa\": container with ID starting with 9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa not found: ID does not exist" containerID="9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa" Sep 08 02:22:46.878985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.878940 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa"} err="failed to get container status \"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa\": rpc error: code = NotFound desc = could not find container \"9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa\": container with ID starting with 9800d8a4ad822c4cfddf487b96596edeaa4eb9a39d58b226a2eb0d6bf25eb6fa not found: ID does not exist" Sep 08 02:22:46.889724 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.889700 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:22:46.895367 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:46.895346 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-670bc3-868557645c-q8qj4"] Sep 08 02:22:47.320842 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:22:47.320805 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60710743-598c-4821-aa9a-67917db6fd52" path="/var/lib/kubelet/pods/60710743-598c-4821-aa9a-67917db6fd52/volumes" Sep 08 02:23:03.121241 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121200 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:23:03.121847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121829 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" containerName="maas-api" Sep 08 02:23:03.121847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121848 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" containerName="maas-api" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121881 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" containerName="mt-curl-7682-30b56d" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121890 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" containerName="mt-curl-7682-30b56d" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121905 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" containerName="maas-api" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121914 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" containerName="maas-api" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121928 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e2ab18-d48d-4255-a04e-f74344614ed4" containerName="payload-pre-processing" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121937 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e2ab18-d48d-4255-a04e-f74344614ed4" containerName="payload-pre-processing" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121948 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433fe496-2675-4dcf-9d52-22d661a50b1e" containerName="payload-processing" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121957 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="433fe496-2675-4dcf-9d52-22d661a50b1e" containerName="payload-processing" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121969 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60710743-598c-4821-aa9a-67917db6fd52" containerName="maas-api" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121977 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60710743-598c-4821-aa9a-67917db6fd52" containerName="maas-api" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121986 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" containerName="payload-pre-processing" Sep 08 02:23:03.122014 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.121995 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" containerName="payload-pre-processing" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122120 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b9669a6-0f72-44f6-8c32-84eaf03a2e11" containerName="maas-api" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122133 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb86ba0-cb9d-40c3-a2c2-572ea7a75659" containerName="maas-api" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122143 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a63279-7e3e-4b16-9ed0-cab1e21700a0" containerName="payload-pre-processing" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122155 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60710743-598c-4821-aa9a-67917db6fd52" containerName="maas-api" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122164 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="433fe496-2675-4dcf-9d52-22d661a50b1e" containerName="payload-processing" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122172 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b366c1a0-8bfd-4659-bb79-6eade8d87e08" containerName="mt-curl-7682-30b56d" Sep 08 02:23:03.122655 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.122181 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e2ab18-d48d-4255-a04e-f74344614ed4" containerName="payload-pre-processing" Sep 08 02:23:03.127267 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.127242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.129540 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.129518 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b23439-serving-cert\"" Sep 08 02:23:03.137233 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.137205 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:23:03.184323 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.184276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.184481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.184380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.184481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.184425 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.184481 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.184479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlwp\" (UniqueName: \"kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.285727 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.285686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.285904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.285740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.285904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.285778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.285904 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.285813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlwp\" (UniqueName: \"kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.286018 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:23:03.285929 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-serving-cert: secret "maas-api-e2e-ipp-a-b23439-serving-cert" not found Sep 08 02:23:03.286018 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:23:03.286008 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls podName:4e6530c8-9968-407b-9fef-7144a144d2f2 nodeName:}" failed. No retries permitted until 2026-09-08 02:23:03.785987081 +0000 UTC m=+1464.963299146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls") pod "maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" (UID: "4e6530c8-9968-407b-9fef-7144a144d2f2") : secret "maas-api-e2e-ipp-a-b23439-serving-cert" not found Sep 08 02:23:03.286635 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.286601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.288224 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.288201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.294041 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.294014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlwp\" (UniqueName: \"kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.732438 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.732399 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:23:03.736335 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.736313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.738626 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.738604 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b23439\"" Sep 08 02:23:03.738734 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.738623 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b23439-dockercfg-vz5cb\"" Sep 08 02:23:03.744419 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.744240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:23:03.769121 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.769089 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:23:03.773116 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.773090 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:03.785023 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.784997 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:23:03.790878 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.790853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7cr\" (UniqueName: \"kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.791021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.790905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.791021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.790986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.793568 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.793543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:03.890147 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.890109 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:23:03.890420 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.890393 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" podUID="617aabf7-df68-4db3-8f77-3574821cb4df" containerName="maas-api" containerID="cri-o://775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254" gracePeriod=30 Sep 08 02:23:03.892349 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.892320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4n6\" (UniqueName: \"kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:03.892467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.892373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.892467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.892450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7cr\" (UniqueName: \"kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.892593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.892523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:03.894065 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.893455 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.901360 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.901333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7cr\" (UniqueName: \"kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr\") pod \"payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:03.993743 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.993665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:03.993743 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.993736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4n6\" (UniqueName: \"kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:03.994583 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:03.994551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:04.006066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.006026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4n6\" (UniqueName: \"kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6\") pod \"payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:04.041448 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.041417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:04.049039 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.048958 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:04.086145 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.085951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:04.167625 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.166517 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:23:04.297636 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.297549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs\") pod \"617aabf7-df68-4db3-8f77-3574821cb4df\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " Sep 08 02:23:04.297636 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.297619 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls\") pod \"617aabf7-df68-4db3-8f77-3574821cb4df\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " Sep 08 02:23:04.297840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.297686 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jngqg\" (UniqueName: \"kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg\") pod \"617aabf7-df68-4db3-8f77-3574821cb4df\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " Sep 08 02:23:04.297840 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.297765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca\") pod \"617aabf7-df68-4db3-8f77-3574821cb4df\" (UID: \"617aabf7-df68-4db3-8f77-3574821cb4df\") " Sep 08 02:23:04.298382 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.298334 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "617aabf7-df68-4db3-8f77-3574821cb4df" (UID: "617aabf7-df68-4db3-8f77-3574821cb4df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:23:04.299800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.299771 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "617aabf7-df68-4db3-8f77-3574821cb4df" (UID: "617aabf7-df68-4db3-8f77-3574821cb4df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:23:04.299930 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.299911 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg" (OuterVolumeSpecName: "kube-api-access-jngqg") pod "617aabf7-df68-4db3-8f77-3574821cb4df" (UID: "617aabf7-df68-4db3-8f77-3574821cb4df"). InnerVolumeSpecName "kube-api-access-jngqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:23:04.300402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.300373 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "617aabf7-df68-4db3-8f77-3574821cb4df" (UID: "617aabf7-df68-4db3-8f77-3574821cb4df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:23:04.350427 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.350398 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:23:04.351660 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:23:04.351634 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6530c8_9968_407b_9fef_7144a144d2f2.slice/crio-c3e20ad2628878a95d5b1c3fc68e9629dcb8c51b6bdc42af85217857dbe1632e WatchSource:0}: Error finding container c3e20ad2628878a95d5b1c3fc68e9629dcb8c51b6bdc42af85217857dbe1632e: Status 404 returned error can't find the container with id c3e20ad2628878a95d5b1c3fc68e9629dcb8c51b6bdc42af85217857dbe1632e Sep 08 02:23:04.362096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.362070 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:23:04.364576 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:23:04.364549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6435b697_0e4f_4d46_8654_5035c7e14dc2.slice/crio-f4bad3ef98a87b4b6ffa2dff15b03591199205a72a1d587a8caeb056ab47cc7a WatchSource:0}: Error finding container f4bad3ef98a87b4b6ffa2dff15b03591199205a72a1d587a8caeb056ab47cc7a: Status 404 returned error can't find the container with id f4bad3ef98a87b4b6ffa2dff15b03591199205a72a1d587a8caeb056ab47cc7a Sep 08 02:23:04.399780 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.399718 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/617aabf7-df68-4db3-8f77-3574821cb4df-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:23:04.399952 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.399808 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:23:04.399952 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.399829 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/617aabf7-df68-4db3-8f77-3574821cb4df-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:23:04.399952 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.399841 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngqg\" (UniqueName: \"kubernetes.io/projected/617aabf7-df68-4db3-8f77-3574821cb4df-kube-api-access-jngqg\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:23:04.413649 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.413508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:23:04.416374 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:23:04.416347 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bd540e_8178_4d8d_91a1_9e5f87ee059e.slice/crio-effa3240e12a13105f63e68a334181f06eb34de3bd3c24e42b4cd7d72abdc194 WatchSource:0}: Error finding container effa3240e12a13105f63e68a334181f06eb34de3bd3c24e42b4cd7d72abdc194: Status 404 returned error can't find the container with id effa3240e12a13105f63e68a334181f06eb34de3bd3c24e42b4cd7d72abdc194 Sep 08 02:23:04.991524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.991487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" event={"ID":"45bd540e-8178-4d8d-91a1-9e5f87ee059e","Type":"ContainerStarted","Data":"d1eee19643c6beb78921fac9fe01b0b0b69596bef5282f8ecea0706d1ac042fc"} Sep 08 02:23:04.991700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.991552 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" event={"ID":"45bd540e-8178-4d8d-91a1-9e5f87ee059e","Type":"ContainerStarted","Data":"effa3240e12a13105f63e68a334181f06eb34de3bd3c24e42b4cd7d72abdc194"} Sep 08 02:23:04.991700 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.991673 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:04.993437 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.993408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" event={"ID":"6435b697-0e4f-4d46-8654-5035c7e14dc2","Type":"ContainerStarted","Data":"67cd3cb889bb4661e8bca9c573ea0a4f43f489d8fe026e4b7279de3ee55cb969"} Sep 08 02:23:04.993608 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.993584 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" event={"ID":"6435b697-0e4f-4d46-8654-5035c7e14dc2","Type":"ContainerStarted","Data":"f4bad3ef98a87b4b6ffa2dff15b03591199205a72a1d587a8caeb056ab47cc7a"} Sep 08 02:23:04.993728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.993618 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:04.994992 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.994967 2559 generic.go:356] "Generic (PLEG): container finished" podID="617aabf7-df68-4db3-8f77-3574821cb4df" containerID="775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254" exitCode=0 Sep 08 02:23:04.995208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.995007 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" event={"ID":"617aabf7-df68-4db3-8f77-3574821cb4df","Type":"ContainerDied","Data":"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254"} Sep 08 02:23:04.995208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.995057 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" event={"ID":"617aabf7-df68-4db3-8f77-3574821cb4df","Type":"ContainerDied","Data":"52453d2ce81010420a55db0d7b3fa048199cb9236bb6a367611baceb9a9e75b2"} Sep 08 02:23:04.995208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.995062 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg" Sep 08 02:23:04.995208 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.995087 2559 scope.go:117] "RemoveContainer" containerID="775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254" Sep 08 02:23:04.997199 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.997171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" event={"ID":"4e6530c8-9968-407b-9fef-7144a144d2f2","Type":"ContainerStarted","Data":"811b0a6350aeda811a0031a2ac3f74839a0f6d04b80bd408ce749e49d1f26c84"} Sep 08 02:23:04.997284 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.997212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" event={"ID":"4e6530c8-9968-407b-9fef-7144a144d2f2","Type":"ContainerStarted","Data":"c3e20ad2628878a95d5b1c3fc68e9629dcb8c51b6bdc42af85217857dbe1632e"} Sep 08 02:23:04.997446 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:04.997423 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:05.006989 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.006940 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" podStartSLOduration=2.00692205 podStartE2EDuration="2.00692205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:05.006201197 +0000 UTC m=+1466.183513270" watchObservedRunningTime="2026-09-08 02:23:05.00692205 +0000 UTC m=+1466.184234124" Sep 08 02:23:05.007673 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.007653 2559 scope.go:117] "RemoveContainer" containerID="775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254" Sep 08 02:23:05.007952 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:23:05.007931 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254\": container with ID starting with 775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254 not found: ID does not exist" containerID="775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254" Sep 08 02:23:05.008021 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.007967 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254"} err="failed to get container status \"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254\": rpc error: code = NotFound desc = could not find container \"775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254\": container with ID starting with 775a2858ccb89a8cd518462bfc4106edbeb347e4cb7b3e367119322640cce254 not found: ID does not exist" Sep 08 02:23:05.020929 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.020885 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" podStartSLOduration=2.020868103 podStartE2EDuration="2.020868103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:05.020565186 +0000 UTC m=+1466.197877260" watchObservedRunningTime="2026-09-08 02:23:05.020868103 +0000 UTC m=+1466.198180175" Sep 08 02:23:05.037038 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.036958 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" podStartSLOduration=1.6064386750000001 podStartE2EDuration="2.036946623s" totalImagesPullingTime="430.507948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:05.03469049 +0000 UTC m=+1466.212002563" watchObservedRunningTime="2026-09-08 02:23:05.036946623 +0000 UTC m=+1466.214258695" Sep 08 02:23:05.052027 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.052005 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:23:05.055888 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.055866 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d1413-6b5454555-j6sxg"] Sep 08 02:23:05.320732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:05.320657 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617aabf7-df68-4db3-8f77-3574821cb4df" path="/var/lib/kubelet/pods/617aabf7-df68-4db3-8f77-3574821cb4df/volumes" Sep 08 02:23:11.009586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:11.009555 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:23:16.006725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:16.006684 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:23:16.007178 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:16.006756 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:23:32.375068 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.375026 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:23:32.375977 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.375952 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617aabf7-df68-4db3-8f77-3574821cb4df" containerName="maas-api" Sep 08 02:23:32.376174 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.376159 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617aabf7-df68-4db3-8f77-3574821cb4df" containerName="maas-api" Sep 08 02:23:32.376547 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.376522 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="617aabf7-df68-4db3-8f77-3574821cb4df" containerName="maas-api" Sep 08 02:23:32.380046 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.380020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.382373 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.382350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d4845d-serving-cert\"" Sep 08 02:23:32.391460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.391429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:23:32.563466 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.563433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.563630 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.563487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9j6\" (UniqueName: \"kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.563681 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.563629 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.563728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.563699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.664935 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.664898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.665117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.664944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9j6\" (UniqueName: \"kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.665117 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:23:32.665039 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-serving-cert: secret "maas-api-e2e-ipp-b-d4845d-serving-cert" not found Sep 08 02:23:32.665117 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:23:32.665098 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls podName:1fefb0c0-96b0-4728-84c6-4b38cd735330 nodeName:}" failed. No retries permitted until 2026-09-08 02:23:33.165078408 +0000 UTC m=+1494.342390471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls") pod "maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" (UID: "1fefb0c0-96b0-4728-84c6-4b38cd735330") : secret "maas-api-e2e-ipp-b-d4845d-serving-cert" not found Sep 08 02:23:32.665317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.665120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.665317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.665172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.665703 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.665681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.667616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.667588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:32.672718 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:32.672700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9j6\" (UniqueName: \"kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:33.169892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:33.169851 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:33.172443 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:33.172412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:33.303975 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:33.303905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:33.586523 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:33.586492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:23:33.588071 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:23:33.588048 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fefb0c0_96b0_4728_84c6_4b38cd735330.slice/crio-a0deba357f199f8b09c143246eba8ce6f455798c0057db00510179f35b7a913c WatchSource:0}: Error finding container a0deba357f199f8b09c143246eba8ce6f455798c0057db00510179f35b7a913c: Status 404 returned error can't find the container with id a0deba357f199f8b09c143246eba8ce6f455798c0057db00510179f35b7a913c Sep 08 02:23:34.188134 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:34.188096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" event={"ID":"1fefb0c0-96b0-4728-84c6-4b38cd735330","Type":"ContainerStarted","Data":"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496"} Sep 08 02:23:34.188275 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:34.188152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" event={"ID":"1fefb0c0-96b0-4728-84c6-4b38cd735330","Type":"ContainerStarted","Data":"a0deba357f199f8b09c143246eba8ce6f455798c0057db00510179f35b7a913c"} Sep 08 02:23:34.188275 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:34.188211 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:23:34.204501 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:34.204456 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" podStartSLOduration=1.7924214379999999 podStartE2EDuration="2.204443723s" totalImagesPullingTime="412.022285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:34.202538245 +0000 UTC m=+1495.379850317" watchObservedRunningTime="2026-09-08 02:23:34.204443723 +0000 UTC m=+1495.381755796" Sep 08 02:23:39.432442 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:39.432413 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:23:39.432939 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:39.432917 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:23:40.199516 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:23:40.199488 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:24:01.340395 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:01.340364 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:24:39.652821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:39.652795 2559 scope.go:117] "RemoveContainer" containerID="855af113854895debbb79c7efc42be06dd5bdc4d9a30e1b8286b0854e78dd3e4" Sep 08 02:24:41.126520 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:41.126489 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6_45bd540e-8178-4d8d-91a1-9e5f87ee059e/payload-processing/0.log" Sep 08 02:24:41.241195 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:41.241157 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:24:45.841570 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.841537 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:24:45.845445 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.845422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:45.847810 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.847785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ae715d-serving-cert\"" Sep 08 02:24:45.857064 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.857039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:24:45.979714 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.979680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:45.979900 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.979749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:45.979900 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.979806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qm6\" (UniqueName: \"kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:45.979976 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:45.979927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.080454 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.080418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.080454 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.080459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.080680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.080517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.080680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.080537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qm6\" (UniqueName: \"kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.081220 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.081195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.082987 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.082961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.083095 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.083072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.088177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.088146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qm6\" (UniqueName: \"kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6\") pod \"maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.157099 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.157071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:46.429695 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.429670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:24:46.431461 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:24:46.431436 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3160b18a_a73d_4a63_90fc_38c790281e15.slice/crio-824ff43e8144c7eafb848d1bf8cc3f9dad1ac1339b60716af0be13235b77f00d WatchSource:0}: Error finding container 824ff43e8144c7eafb848d1bf8cc3f9dad1ac1339b60716af0be13235b77f00d: Status 404 returned error can't find the container with id 824ff43e8144c7eafb848d1bf8cc3f9dad1ac1339b60716af0be13235b77f00d Sep 08 02:24:46.633285 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:46.633248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" event={"ID":"3160b18a-a73d-4a63-90fc-38c790281e15","Type":"ContainerStarted","Data":"824ff43e8144c7eafb848d1bf8cc3f9dad1ac1339b60716af0be13235b77f00d"} Sep 08 02:24:47.640747 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:47.640711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" event={"ID":"3160b18a-a73d-4a63-90fc-38c790281e15","Type":"ContainerStarted","Data":"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43"} Sep 08 02:24:47.641135 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:47.640887 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:24:47.657489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:47.657445 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" podStartSLOduration=2.124003629 podStartE2EDuration="2.657432274s" totalImagesPullingTime="533.428645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:24:47.656579894 +0000 UTC m=+1568.833891968" watchObservedRunningTime="2026-09-08 02:24:47.657432274 +0000 UTC m=+1568.834744345" Sep 08 02:24:53.652332 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:24:53.652305 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:25:16.389050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.389015 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:25:16.389642 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.389263 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" podUID="3160b18a-a73d-4a63-90fc-38c790281e15" containerName="maas-api" containerID="cri-o://8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43" gracePeriod=30 Sep 08 02:25:16.646592 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.646533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:25:16.788960 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.788927 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls\") pod \"3160b18a-a73d-4a63-90fc-38c790281e15\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " Sep 08 02:25:16.789137 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.789008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca\") pod \"3160b18a-a73d-4a63-90fc-38c790281e15\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " Sep 08 02:25:16.789137 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.789069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs\") pod \"3160b18a-a73d-4a63-90fc-38c790281e15\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " Sep 08 02:25:16.789137 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.789115 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5qm6\" (UniqueName: \"kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6\") pod \"3160b18a-a73d-4a63-90fc-38c790281e15\" (UID: \"3160b18a-a73d-4a63-90fc-38c790281e15\") " Sep 08 02:25:16.789409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.789387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3160b18a-a73d-4a63-90fc-38c790281e15" (UID: "3160b18a-a73d-4a63-90fc-38c790281e15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:25:16.791218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.791183 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3160b18a-a73d-4a63-90fc-38c790281e15" (UID: "3160b18a-a73d-4a63-90fc-38c790281e15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:25:16.791218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.791203 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3160b18a-a73d-4a63-90fc-38c790281e15" (UID: "3160b18a-a73d-4a63-90fc-38c790281e15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:25:16.791416 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.791188 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6" (OuterVolumeSpecName: "kube-api-access-m5qm6") pod "3160b18a-a73d-4a63-90fc-38c790281e15" (UID: "3160b18a-a73d-4a63-90fc-38c790281e15"). InnerVolumeSpecName "kube-api-access-m5qm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:25:16.823073 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.823043 2559 generic.go:356] "Generic (PLEG): container finished" podID="3160b18a-a73d-4a63-90fc-38c790281e15" containerID="8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43" exitCode=0 Sep 08 02:25:16.823221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.823092 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" Sep 08 02:25:16.823221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.823127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" event={"ID":"3160b18a-a73d-4a63-90fc-38c790281e15","Type":"ContainerDied","Data":"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43"} Sep 08 02:25:16.823221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.823185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr" event={"ID":"3160b18a-a73d-4a63-90fc-38c790281e15","Type":"ContainerDied","Data":"824ff43e8144c7eafb848d1bf8cc3f9dad1ac1339b60716af0be13235b77f00d"} Sep 08 02:25:16.823221 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.823213 2559 scope.go:117] "RemoveContainer" containerID="8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43" Sep 08 02:25:16.833602 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.833583 2559 scope.go:117] "RemoveContainer" containerID="8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43" Sep 08 02:25:16.833890 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:25:16.833874 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43\": container with ID starting with 8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43 not found: ID does not exist" containerID="8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43" Sep 08 02:25:16.833935 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.833899 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43"} err="failed to get container status \"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43\": rpc error: code = NotFound desc = could not find container \"8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43\": container with ID starting with 8475d898295746f81be742f7cf5178b3a5e15c53b42e05d69e96359e2a87be43 not found: ID does not exist" Sep 08 02:25:16.844232 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.844207 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:25:16.846711 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.846689 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae715d-668dc5998d-glzxr"] Sep 08 02:25:16.890216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.890185 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160b18a-a73d-4a63-90fc-38c790281e15-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:16.890216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.890211 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:16.890216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.890220 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qm6\" (UniqueName: \"kubernetes.io/projected/3160b18a-a73d-4a63-90fc-38c790281e15-kube-api-access-m5qm6\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:16.890444 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:16.890228 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160b18a-a73d-4a63-90fc-38c790281e15-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:17.320557 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:17.320520 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3160b18a-a73d-4a63-90fc-38c790281e15" path="/var/lib/kubelet/pods/3160b18a-a73d-4a63-90fc-38c790281e15/volumes" Sep 08 02:25:41.788214 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.788179 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:25:41.788729 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.788432 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" podUID="4e6530c8-9968-407b-9fef-7144a144d2f2" containerName="maas-api" containerID="cri-o://811b0a6350aeda811a0031a2ac3f74839a0f6d04b80bd408ce749e49d1f26c84" gracePeriod=30 Sep 08 02:25:41.844413 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.844381 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:25:41.844642 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.844611 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" podUID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" containerName="payload-processing" containerID="cri-o://d1eee19643c6beb78921fac9fe01b0b0b69596bef5282f8ecea0706d1ac042fc" gracePeriod=30 Sep 08 02:25:41.852831 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.852798 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:25:41.853106 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.853076 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" podUID="6435b697-0e4f-4d46-8654-5035c7e14dc2" containerName="payload-pre-processing" containerID="cri-o://67cd3cb889bb4661e8bca9c573ea0a4f43f489d8fe026e4b7279de3ee55cb969" gracePeriod=30 Sep 08 02:25:41.990880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.990800 2559 generic.go:356] "Generic (PLEG): container finished" podID="4e6530c8-9968-407b-9fef-7144a144d2f2" containerID="811b0a6350aeda811a0031a2ac3f74839a0f6d04b80bd408ce749e49d1f26c84" exitCode=0 Sep 08 02:25:41.990880 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.990867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" event={"ID":"4e6530c8-9968-407b-9fef-7144a144d2f2","Type":"ContainerDied","Data":"811b0a6350aeda811a0031a2ac3f74839a0f6d04b80bd408ce749e49d1f26c84"} Sep 08 02:25:41.993102 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.993072 2559 generic.go:356] "Generic (PLEG): container finished" podID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" containerID="d1eee19643c6beb78921fac9fe01b0b0b69596bef5282f8ecea0706d1ac042fc" exitCode=0 Sep 08 02:25:41.993311 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.993156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" event={"ID":"45bd540e-8178-4d8d-91a1-9e5f87ee059e","Type":"ContainerDied","Data":"d1eee19643c6beb78921fac9fe01b0b0b69596bef5282f8ecea0706d1ac042fc"} Sep 08 02:25:41.995163 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.995052 2559 generic.go:356] "Generic (PLEG): container finished" podID="6435b697-0e4f-4d46-8654-5035c7e14dc2" containerID="67cd3cb889bb4661e8bca9c573ea0a4f43f489d8fe026e4b7279de3ee55cb969" exitCode=0 Sep 08 02:25:41.995163 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:41.995117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" event={"ID":"6435b697-0e4f-4d46-8654-5035c7e14dc2","Type":"ContainerDied","Data":"67cd3cb889bb4661e8bca9c573ea0a4f43f489d8fe026e4b7279de3ee55cb969"} Sep 08 02:25:42.119172 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.119149 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:25:42.175406 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.175380 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:25:42.180647 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.180625 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:25:42.227255 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227221 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume\") pod \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " Sep 08 02:25:42.227475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227263 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwlwp\" (UniqueName: \"kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp\") pod \"4e6530c8-9968-407b-9fef-7144a144d2f2\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " Sep 08 02:25:42.227475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs\") pod \"4e6530c8-9968-407b-9fef-7144a144d2f2\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " Sep 08 02:25:42.227475 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227364 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k4n6\" (UniqueName: \"kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6\") pod \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\" (UID: \"45bd540e-8178-4d8d-91a1-9e5f87ee059e\") " Sep 08 02:25:42.227698 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227672 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45bd540e-8178-4d8d-91a1-9e5f87ee059e" (UID: "45bd540e-8178-4d8d-91a1-9e5f87ee059e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:25:42.227819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq7cr\" (UniqueName: \"kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr\") pod \"6435b697-0e4f-4d46-8654-5035c7e14dc2\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " Sep 08 02:25:42.227881 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227862 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") pod \"4e6530c8-9968-407b-9fef-7144a144d2f2\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " Sep 08 02:25:42.227941 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227924 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca\") pod \"4e6530c8-9968-407b-9fef-7144a144d2f2\" (UID: \"4e6530c8-9968-407b-9fef-7144a144d2f2\") " Sep 08 02:25:42.228003 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.227976 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume\") pod \"6435b697-0e4f-4d46-8654-5035c7e14dc2\" (UID: \"6435b697-0e4f-4d46-8654-5035c7e14dc2\") " Sep 08 02:25:42.228350 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.228327 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd540e-8178-4d8d-91a1-9e5f87ee059e-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.228859 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.228832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6435b697-0e4f-4d46-8654-5035c7e14dc2" (UID: "6435b697-0e4f-4d46-8654-5035c7e14dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:25:42.229227 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.229201 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e6530c8-9968-407b-9fef-7144a144d2f2" (UID: "4e6530c8-9968-407b-9fef-7144a144d2f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:25:42.230026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.229986 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6" (OuterVolumeSpecName: "kube-api-access-4k4n6") pod "45bd540e-8178-4d8d-91a1-9e5f87ee059e" (UID: "45bd540e-8178-4d8d-91a1-9e5f87ee059e"). InnerVolumeSpecName "kube-api-access-4k4n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:25:42.230136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.229997 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e6530c8-9968-407b-9fef-7144a144d2f2" (UID: "4e6530c8-9968-407b-9fef-7144a144d2f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:25:42.230136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.230077 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp" (OuterVolumeSpecName: "kube-api-access-wwlwp") pod "4e6530c8-9968-407b-9fef-7144a144d2f2" (UID: "4e6530c8-9968-407b-9fef-7144a144d2f2"). InnerVolumeSpecName "kube-api-access-wwlwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:25:42.230526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.230499 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e6530c8-9968-407b-9fef-7144a144d2f2" (UID: "4e6530c8-9968-407b-9fef-7144a144d2f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:25:42.230658 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.230642 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr" (OuterVolumeSpecName: "kube-api-access-sq7cr") pod "6435b697-0e4f-4d46-8654-5035c7e14dc2" (UID: "6435b697-0e4f-4d46-8654-5035c7e14dc2"). InnerVolumeSpecName "kube-api-access-sq7cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:25:42.329149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329111 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6530c8-9968-407b-9fef-7144a144d2f2-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329142 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b697-0e4f-4d46-8654-5035c7e14dc2-plugins-config-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329151 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlwp\" (UniqueName: \"kubernetes.io/projected/4e6530c8-9968-407b-9fef-7144a144d2f2-kube-api-access-wwlwp\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329161 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329171 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k4n6\" (UniqueName: \"kubernetes.io/projected/45bd540e-8178-4d8d-91a1-9e5f87ee059e-kube-api-access-4k4n6\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329179 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7cr\" (UniqueName: \"kubernetes.io/projected/6435b697-0e4f-4d46-8654-5035c7e14dc2-kube-api-access-sq7cr\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:42.329415 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:42.329187 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e6530c8-9968-407b-9fef-7144a144d2f2-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:25:43.002136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.002104 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" Sep 08 02:25:43.002576 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.002132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf" event={"ID":"4e6530c8-9968-407b-9fef-7144a144d2f2","Type":"ContainerDied","Data":"c3e20ad2628878a95d5b1c3fc68e9629dcb8c51b6bdc42af85217857dbe1632e"} Sep 08 02:25:43.002576 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.002202 2559 scope.go:117] "RemoveContainer" containerID="811b0a6350aeda811a0031a2ac3f74839a0f6d04b80bd408ce749e49d1f26c84" Sep 08 02:25:43.003821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.003804 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" Sep 08 02:25:43.003821 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.003808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6" event={"ID":"45bd540e-8178-4d8d-91a1-9e5f87ee059e","Type":"ContainerDied","Data":"effa3240e12a13105f63e68a334181f06eb34de3bd3c24e42b4cd7d72abdc194"} Sep 08 02:25:43.005892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.005726 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" event={"ID":"6435b697-0e4f-4d46-8654-5035c7e14dc2","Type":"ContainerDied","Data":"f4bad3ef98a87b4b6ffa2dff15b03591199205a72a1d587a8caeb056ab47cc7a"} Sep 08 02:25:43.005892 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.005811 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc" Sep 08 02:25:43.023207 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.023183 2559 scope.go:117] "RemoveContainer" containerID="d1eee19643c6beb78921fac9fe01b0b0b69596bef5282f8ecea0706d1ac042fc" Sep 08 02:25:43.030403 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.030379 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:25:43.033650 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.033635 2559 scope.go:117] "RemoveContainer" containerID="67cd3cb889bb4661e8bca9c573ea0a4f43f489d8fe026e4b7279de3ee55cb969" Sep 08 02:25:43.034814 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.034723 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b23439-67bfbb9767-vr8rf"] Sep 08 02:25:43.043780 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.043762 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:25:43.051442 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.051420 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b23439-647fdb4d8b-jjcwc"] Sep 08 02:25:43.062778 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.062715 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:25:43.066730 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.066706 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b23439-7c56fd7cf-86tv6"] Sep 08 02:25:43.319726 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.319647 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" path="/var/lib/kubelet/pods/45bd540e-8178-4d8d-91a1-9e5f87ee059e/volumes" Sep 08 02:25:43.320026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.320013 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6530c8-9968-407b-9fef-7144a144d2f2" path="/var/lib/kubelet/pods/4e6530c8-9968-407b-9fef-7144a144d2f2/volumes" Sep 08 02:25:43.320351 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:25:43.320338 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6435b697-0e4f-4d46-8654-5035c7e14dc2" path="/var/lib/kubelet/pods/6435b697-0e4f-4d46-8654-5035c7e14dc2/volumes" Sep 08 02:26:46.675963 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:46.675930 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:26:46.676409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:46.676126 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" podUID="1fefb0c0-96b0-4728-84c6-4b38cd735330" containerName="maas-api" containerID="cri-o://fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496" gracePeriod=30 Sep 08 02:26:46.923812 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:46.923791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:26:47.026985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.026911 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc9j6\" (UniqueName: \"kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6\") pod \"1fefb0c0-96b0-4728-84c6-4b38cd735330\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " Sep 08 02:26:47.026985 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.026953 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs\") pod \"1fefb0c0-96b0-4728-84c6-4b38cd735330\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " Sep 08 02:26:47.027156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.027112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") pod \"1fefb0c0-96b0-4728-84c6-4b38cd735330\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " Sep 08 02:26:47.027201 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.027163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca\") pod \"1fefb0c0-96b0-4728-84c6-4b38cd735330\" (UID: \"1fefb0c0-96b0-4728-84c6-4b38cd735330\") " Sep 08 02:26:47.027605 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.027578 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1fefb0c0-96b0-4728-84c6-4b38cd735330" (UID: "1fefb0c0-96b0-4728-84c6-4b38cd735330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:26:47.029095 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.029066 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1fefb0c0-96b0-4728-84c6-4b38cd735330" (UID: "1fefb0c0-96b0-4728-84c6-4b38cd735330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:26:47.029218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.029128 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6" (OuterVolumeSpecName: "kube-api-access-cc9j6") pod "1fefb0c0-96b0-4728-84c6-4b38cd735330" (UID: "1fefb0c0-96b0-4728-84c6-4b38cd735330"). InnerVolumeSpecName "kube-api-access-cc9j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:26:47.029218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.029168 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1fefb0c0-96b0-4728-84c6-4b38cd735330" (UID: "1fefb0c0-96b0-4728-84c6-4b38cd735330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:26:47.128178 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.128149 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:26:47.128178 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.128172 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fefb0c0-96b0-4728-84c6-4b38cd735330-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:26:47.128178 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.128182 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9j6\" (UniqueName: \"kubernetes.io/projected/1fefb0c0-96b0-4728-84c6-4b38cd735330-kube-api-access-cc9j6\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:26:47.128527 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.128191 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fefb0c0-96b0-4728-84c6-4b38cd735330-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:26:47.404128 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.404092 2559 generic.go:356] "Generic (PLEG): container finished" podID="1fefb0c0-96b0-4728-84c6-4b38cd735330" containerID="fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496" exitCode=0 Sep 08 02:26:47.404277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.404128 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" event={"ID":"1fefb0c0-96b0-4728-84c6-4b38cd735330","Type":"ContainerDied","Data":"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496"} Sep 08 02:26:47.404277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.404185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" event={"ID":"1fefb0c0-96b0-4728-84c6-4b38cd735330","Type":"ContainerDied","Data":"a0deba357f199f8b09c143246eba8ce6f455798c0057db00510179f35b7a913c"} Sep 08 02:26:47.404277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.404206 2559 scope.go:117] "RemoveContainer" containerID="fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496" Sep 08 02:26:47.404277 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.404138 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz" Sep 08 02:26:47.413151 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.413134 2559 scope.go:117] "RemoveContainer" containerID="fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496" Sep 08 02:26:47.413440 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:26:47.413420 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496\": container with ID starting with fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496 not found: ID does not exist" containerID="fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496" Sep 08 02:26:47.413498 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.413448 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496"} err="failed to get container status \"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496\": rpc error: code = NotFound desc = could not find container \"fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496\": container with ID starting with fc81e619f037b1e007726740e878fc12898805a64e947d452b51425fd4ed8496 not found: ID does not exist" Sep 08 02:26:47.425276 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.425255 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:26:47.429330 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:47.429309 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4845d-7bf889d788-7pjwz"] Sep 08 02:26:49.319888 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:26:49.319855 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fefb0c0-96b0-4728-84c6-4b38cd735330" path="/var/lib/kubelet/pods/1fefb0c0-96b0-4728-84c6-4b38cd735330/volumes" Sep 08 02:27:04.202105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.202014 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:27:04.202590 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.202255 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" podUID="0899785a-959b-4612-b214-addb714eec98" containerName="maas-api" containerID="cri-o://2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d" gracePeriod=30 Sep 08 02:27:04.454801 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.454742 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:27:04.487601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.487575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbkkg\" (UniqueName: \"kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg\") pod \"0899785a-959b-4612-b214-addb714eec98\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " Sep 08 02:27:04.487601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.487610 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs\") pod \"0899785a-959b-4612-b214-addb714eec98\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " Sep 08 02:27:04.487793 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.487649 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") pod \"0899785a-959b-4612-b214-addb714eec98\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " Sep 08 02:27:04.489706 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.489680 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0899785a-959b-4612-b214-addb714eec98" (UID: "0899785a-959b-4612-b214-addb714eec98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:04.490127 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.490094 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0899785a-959b-4612-b214-addb714eec98" (UID: "0899785a-959b-4612-b214-addb714eec98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:04.490218 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.490191 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg" (OuterVolumeSpecName: "kube-api-access-mbkkg") pod "0899785a-959b-4612-b214-addb714eec98" (UID: "0899785a-959b-4612-b214-addb714eec98"). InnerVolumeSpecName "kube-api-access-mbkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:27:04.513162 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.513132 2559 generic.go:356] "Generic (PLEG): container finished" podID="0899785a-959b-4612-b214-addb714eec98" containerID="2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d" exitCode=0 Sep 08 02:27:04.513272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.513175 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" Sep 08 02:27:04.513272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.513170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" event={"ID":"0899785a-959b-4612-b214-addb714eec98","Type":"ContainerDied","Data":"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d"} Sep 08 02:27:04.513389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.513308 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc" event={"ID":"0899785a-959b-4612-b214-addb714eec98","Type":"ContainerDied","Data":"adc76d4c75b9759a4f958ee0cfe109b2d6bc259415145ee5e699a42a24936ae3"} Sep 08 02:27:04.513389 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.513329 2559 scope.go:117] "RemoveContainer" containerID="2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d" Sep 08 02:27:04.522240 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.522224 2559 scope.go:117] "RemoveContainer" containerID="2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d" Sep 08 02:27:04.522522 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:27:04.522505 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d\": container with ID starting with 2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d not found: ID does not exist" containerID="2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d" Sep 08 02:27:04.522593 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.522534 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d"} err="failed to get container status \"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d\": rpc error: code = NotFound desc = could not find container \"2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d\": container with ID starting with 2f76dc62aea18b562d9a4bc8da610f814c97b4cdd0cbb981690a3ddff7cc291d not found: ID does not exist" Sep 08 02:27:04.588995 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.588960 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca\") pod \"0899785a-959b-4612-b214-addb714eec98\" (UID: \"0899785a-959b-4612-b214-addb714eec98\") " Sep 08 02:27:04.589354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.589334 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkkg\" (UniqueName: \"kubernetes.io/projected/0899785a-959b-4612-b214-addb714eec98-kube-api-access-mbkkg\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:04.589354 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.589353 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:04.589518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.589362 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0899785a-959b-4612-b214-addb714eec98-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:04.589518 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.589424 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0899785a-959b-4612-b214-addb714eec98" (UID: "0899785a-959b-4612-b214-addb714eec98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:27:04.690133 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.690102 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0899785a-959b-4612-b214-addb714eec98-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:04.839375 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.839344 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:27:04.843004 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:04.842979 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fcdd46-66946cc89b-gt7bc"] Sep 08 02:27:05.320118 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:05.320082 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0899785a-959b-4612-b214-addb714eec98" path="/var/lib/kubelet/pods/0899785a-959b-4612-b214-addb714eec98/volumes" Sep 08 02:27:11.086907 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:11.086873 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:27:11.089428 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:11.087097 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" containerID="cri-o://27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1" gracePeriod=30 Sep 08 02:27:16.350673 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.350644 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:27:16.380272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380169 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ggr\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380272 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380355 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380399 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380482 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380526 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380520 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380545 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.380897 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.380595 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs\") pod \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\" (UID: \"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2\") " Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.381242 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.381463 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-credential-socket\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.381517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.381726 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.382044 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:27:16.383565 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.382204 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:27:16.384035 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.383878 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr" (OuterVolumeSpecName: "kube-api-access-45ggr") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "kube-api-access-45ggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:27:16.384746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.384681 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 02:27:16.385026 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.385003 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:27:16.385310 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.385268 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" (UID: "7856e7b2-150e-4979-97f0-6fdcd2b3f1c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:27:16.482362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482332 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-token\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482362 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482359 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istiod-ca-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482371 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-podinfo\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482382 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-data\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482395 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-socket\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482407 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-workload-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482419 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ggr\" (UniqueName: \"kubernetes.io/projected/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-kube-api-access-45ggr\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.482572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.482430 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2-istio-envoy\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:16.584403 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.584370 2559 generic.go:356] "Generic (PLEG): container finished" podID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerID="27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1" exitCode=0 Sep 08 02:27:16.584555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.584423 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" Sep 08 02:27:16.584555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.584423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerDied","Data":"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1"} Sep 08 02:27:16.584555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.584489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm" event={"ID":"7856e7b2-150e-4979-97f0-6fdcd2b3f1c2","Type":"ContainerDied","Data":"a24c0e64030905c32147d491ae22f152a62d57811c32dab1d6a75133e4748e0a"} Sep 08 02:27:16.584555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.584535 2559 scope.go:117] "RemoveContainer" containerID="27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1" Sep 08 02:27:16.594319 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.594279 2559 scope.go:117] "RemoveContainer" containerID="7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90" Sep 08 02:27:16.602847 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.602827 2559 scope.go:117] "RemoveContainer" containerID="27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1" Sep 08 02:27:16.603101 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:27:16.603081 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1\": container with ID starting with 27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1 not found: ID does not exist" containerID="27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1" Sep 08 02:27:16.603156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.603110 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1"} err="failed to get container status \"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1\": rpc error: code = NotFound desc = could not find container \"27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1\": container with ID starting with 27a9c3e72550c3899f81105be67db092cafaeb7acdef8bc9f0c6b1d499fc1ad1 not found: ID does not exist" Sep 08 02:27:16.603156 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.603128 2559 scope.go:117] "RemoveContainer" containerID="7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90" Sep 08 02:27:16.603365 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:27:16.603345 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90\": container with ID starting with 7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90 not found: ID does not exist" containerID="7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90" Sep 08 02:27:16.603417 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.603371 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90"} err="failed to get container status \"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90\": rpc error: code = NotFound desc = could not find container \"7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90\": container with ID starting with 7666ca286bcf762e71cc3baadb1fd87e4c54066cf7ac254dbc67878af5bddc90 not found: ID does not exist" Sep 08 02:27:16.608782 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.608757 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:27:16.613164 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:16.613145 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fcdd46-openshift-default-544dfb4d4b-qt8bm"] Sep 08 02:27:17.320819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:17.320785 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" path="/var/lib/kubelet/pods/7856e7b2-150e-4979-97f0-6fdcd2b3f1c2/volumes" Sep 08 02:27:21.942689 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:21.942643 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:27:21.943185 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:21.942869 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" podUID="82fa4997-0ede-4470-bf6f-15042f79f152" containerName="maas-api" containerID="cri-o://3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927" gracePeriod=30 Sep 08 02:27:22.200345 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.200257 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:27:22.234882 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.234847 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs\") pod \"82fa4997-0ede-4470-bf6f-15042f79f152\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " Sep 08 02:27:22.235100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.234903 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls\") pod \"82fa4997-0ede-4470-bf6f-15042f79f152\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " Sep 08 02:27:22.235100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.234923 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j\") pod \"82fa4997-0ede-4470-bf6f-15042f79f152\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " Sep 08 02:27:22.235100 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.234968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca\") pod \"82fa4997-0ede-4470-bf6f-15042f79f152\" (UID: \"82fa4997-0ede-4470-bf6f-15042f79f152\") " Sep 08 02:27:22.235412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.235382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82fa4997-0ede-4470-bf6f-15042f79f152" (UID: "82fa4997-0ede-4470-bf6f-15042f79f152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:27:22.237099 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.237072 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82fa4997-0ede-4470-bf6f-15042f79f152" (UID: "82fa4997-0ede-4470-bf6f-15042f79f152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:22.237194 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.237130 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j" (OuterVolumeSpecName: "kube-api-access-ndp7j") pod "82fa4997-0ede-4470-bf6f-15042f79f152" (UID: "82fa4997-0ede-4470-bf6f-15042f79f152"). InnerVolumeSpecName "kube-api-access-ndp7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:27:22.237194 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.237128 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82fa4997-0ede-4470-bf6f-15042f79f152" (UID: "82fa4997-0ede-4470-bf6f-15042f79f152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:22.336408 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.336370 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:22.336408 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.336407 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fa4997-0ede-4470-bf6f-15042f79f152-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:22.336660 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.336420 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/82fa4997-0ede-4470-bf6f-15042f79f152-kube-api-access-ndp7j\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:22.336660 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.336437 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa4997-0ede-4470-bf6f-15042f79f152-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:22.623457 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.623355 2559 generic.go:356] "Generic (PLEG): container finished" podID="82fa4997-0ede-4470-bf6f-15042f79f152" containerID="3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927" exitCode=0 Sep 08 02:27:22.623457 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.623406 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" Sep 08 02:27:22.623457 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.623438 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" event={"ID":"82fa4997-0ede-4470-bf6f-15042f79f152","Type":"ContainerDied","Data":"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927"} Sep 08 02:27:22.623752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.623504 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh" event={"ID":"82fa4997-0ede-4470-bf6f-15042f79f152","Type":"ContainerDied","Data":"0066d0bc594c289b7501142a6699d2943e0383307884a6d67e63cb1ef0618318"} Sep 08 02:27:22.623752 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.623534 2559 scope.go:117] "RemoveContainer" containerID="3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927" Sep 08 02:27:22.633953 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.633935 2559 scope.go:117] "RemoveContainer" containerID="3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927" Sep 08 02:27:22.634233 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:27:22.634212 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927\": container with ID starting with 3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927 not found: ID does not exist" containerID="3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927" Sep 08 02:27:22.634355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.634241 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927"} err="failed to get container status \"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927\": rpc error: code = NotFound desc = could not find container \"3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927\": container with ID starting with 3af92dece645c4bcf19b17e5e9f4b64cfee20663dbdd4716660ccbf45a82d927 not found: ID does not exist" Sep 08 02:27:22.652316 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.652271 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:27:22.665007 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:22.664977 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-527ac8-559d7f5b64-h7rjh"] Sep 08 02:27:23.319571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:23.319539 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fa4997-0ede-4470-bf6f-15042f79f152" path="/var/lib/kubelet/pods/82fa4997-0ede-4470-bf6f-15042f79f152/volumes" Sep 08 02:27:32.394739 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.394707 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:27:32.395186 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.394906 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5694fc4746-tfdtk" podUID="1f7db392-58ce-4167-bef6-d9c138984d97" containerName="manager" containerID="cri-o://4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5" gracePeriod=30 Sep 08 02:27:32.645467 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.645404 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:27:32.684771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.684732 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f7db392-58ce-4167-bef6-d9c138984d97" containerID="4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5" exitCode=0 Sep 08 02:27:32.684965 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.684816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-tfdtk" Sep 08 02:27:32.684965 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.684851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-tfdtk" event={"ID":"1f7db392-58ce-4167-bef6-d9c138984d97","Type":"ContainerDied","Data":"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5"} Sep 08 02:27:32.684965 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.684907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-tfdtk" event={"ID":"1f7db392-58ce-4167-bef6-d9c138984d97","Type":"ContainerDied","Data":"81849ac1591120ecfc56cbcc5015ae8e6e24bc554d961e5ae4bbb2ff5ac9b1b2"} Sep 08 02:27:32.684965 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.684933 2559 scope.go:117] "RemoveContainer" containerID="4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5" Sep 08 02:27:32.695478 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.695449 2559 scope.go:117] "RemoveContainer" containerID="4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5" Sep 08 02:27:32.695848 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:27:32.695828 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5\": container with ID starting with 4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5 not found: ID does not exist" containerID="4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5" Sep 08 02:27:32.695923 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.695860 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5"} err="failed to get container status \"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5\": rpc error: code = NotFound desc = could not find container \"4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5\": container with ID starting with 4a0febccf203f5b1ff2fe0b24ce39b4505243be7cf3327084eb3c2cd97f785f5 not found: ID does not exist" Sep 08 02:27:32.736546 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.736509 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") pod \"1f7db392-58ce-4167-bef6-d9c138984d97\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " Sep 08 02:27:32.736728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.736600 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs\") pod \"1f7db392-58ce-4167-bef6-d9c138984d97\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " Sep 08 02:27:32.736728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.736644 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdkq5\" (UniqueName: \"kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5\") pod \"1f7db392-58ce-4167-bef6-d9c138984d97\" (UID: \"1f7db392-58ce-4167-bef6-d9c138984d97\") " Sep 08 02:27:32.738770 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.738728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5" (OuterVolumeSpecName: "kube-api-access-zdkq5") pod "1f7db392-58ce-4167-bef6-d9c138984d97" (UID: "1f7db392-58ce-4167-bef6-d9c138984d97"). InnerVolumeSpecName "kube-api-access-zdkq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:27:32.738770 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.738754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f7db392-58ce-4167-bef6-d9c138984d97" (UID: "1f7db392-58ce-4167-bef6-d9c138984d97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:32.738940 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.738819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1f7db392-58ce-4167-bef6-d9c138984d97" (UID: "1f7db392-58ce-4167-bef6-d9c138984d97"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:27:32.837489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.837447 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:32.837489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.837482 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkq5\" (UniqueName: \"kubernetes.io/projected/1f7db392-58ce-4167-bef6-d9c138984d97-kube-api-access-zdkq5\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:32.837489 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:32.837492 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f7db392-58ce-4167-bef6-d9c138984d97-webhook-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:27:33.010918 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:33.010887 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:27:33.017303 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:33.017265 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5694fc4746-tfdtk"] Sep 08 02:27:33.319903 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:33.319820 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7db392-58ce-4167-bef6-d9c138984d97" path="/var/lib/kubelet/pods/1f7db392-58ce-4167-bef6-d9c138984d97/volumes" Sep 08 02:27:39.710491 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:27:39.710453 2559 scope.go:117] "RemoveContainer" containerID="42bc2829ad29ab491739d16641dbcc7f39333608cb527b39274ee55f23220d20" Sep 08 02:28:33.789505 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789429 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789801 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6435b697-0e4f-4d46-8654-5035c7e14dc2" containerName="payload-pre-processing" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789813 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6435b697-0e4f-4d46-8654-5035c7e14dc2" containerName="payload-pre-processing" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789823 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" containerName="payload-processing" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789828 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" containerName="payload-processing" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789835 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0899785a-959b-4612-b214-addb714eec98" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789841 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0899785a-959b-4612-b214-addb714eec98" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789852 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789857 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789862 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3160b18a-a73d-4a63-90fc-38c790281e15" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789867 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3160b18a-a73d-4a63-90fc-38c790281e15" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789877 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7db392-58ce-4167-bef6-d9c138984d97" containerName="manager" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789881 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7db392-58ce-4167-bef6-d9c138984d97" containerName="manager" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789887 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6530c8-9968-407b-9fef-7144a144d2f2" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789892 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6530c8-9968-407b-9fef-7144a144d2f2" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789900 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789907 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789913 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fefb0c0-96b0-4728-84c6-4b38cd735330" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789918 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fefb0c0-96b0-4728-84c6-4b38cd735330" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789925 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa4997-0ede-4470-bf6f-15042f79f152" containerName="maas-api" Sep 08 02:28:33.789919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789930 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa4997-0ede-4470-bf6f-15042f79f152" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789984 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789992 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6435b697-0e4f-4d46-8654-5035c7e14dc2" containerName="payload-pre-processing" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.789999 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa4997-0ede-4470-bf6f-15042f79f152" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790006 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fefb0c0-96b0-4728-84c6-4b38cd735330" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790011 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3160b18a-a73d-4a63-90fc-38c790281e15" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790017 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6530c8-9968-407b-9fef-7144a144d2f2" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790023 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bd540e-8178-4d8d-91a1-9e5f87ee059e" containerName="payload-processing" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790029 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0899785a-959b-4612-b214-addb714eec98" containerName="maas-api" Sep 08 02:28:33.790617 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.790034 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7db392-58ce-4167-bef6-d9c138984d97" containerName="manager" Sep 08 02:28:33.793064 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.793049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.796265 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.796244 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 02:28:33.796409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.796311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 02:28:33.796409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.796328 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc7lc\"" Sep 08 02:28:33.796409 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.796244 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 02:28:33.802008 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.801986 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:33.880575 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.880540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.880575 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.880574 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vvg\" (UniqueName: \"kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.880785 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.880660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.981259 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.981230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.981431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.981325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.981431 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.981352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vvg\" (UniqueName: \"kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.983819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.983791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.983941 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.983791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:33.989861 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:33.989837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vvg\" (UniqueName: \"kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg\") pod \"maas-controller-5694fc4746-8jbl4\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:34.104452 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:34.104394 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:34.378056 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:34.378026 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:34.379444 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:28:34.379420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c62c84_4899_4e91_b3d2_b97263197dcc.slice/crio-9f493d7df136deeaac64bab67eeb15dedd24f6fabc049d9f2607a030e20de6fc WatchSource:0}: Error finding container 9f493d7df136deeaac64bab67eeb15dedd24f6fabc049d9f2607a030e20de6fc: Status 404 returned error can't find the container with id 9f493d7df136deeaac64bab67eeb15dedd24f6fabc049d9f2607a030e20de6fc Sep 08 02:28:34.380761 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:34.380744 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 02:28:35.034914 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:35.034878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-8jbl4" event={"ID":"07c62c84-4899-4e91-b3d2-b97263197dcc","Type":"ContainerStarted","Data":"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce"} Sep 08 02:28:35.035271 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:35.034930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-8jbl4" event={"ID":"07c62c84-4899-4e91-b3d2-b97263197dcc","Type":"ContainerStarted","Data":"9f493d7df136deeaac64bab67eeb15dedd24f6fabc049d9f2607a030e20de6fc"} Sep 08 02:28:35.035271 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:35.035012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:35.055107 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:35.055050 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5694fc4746-8jbl4" podStartSLOduration=1.5589894850000001 podStartE2EDuration="2.055035158s" totalImagesPullingTime="496.045673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:28:35.052370674 +0000 UTC m=+1796.229682745" watchObservedRunningTime="2026-09-08 02:28:35.055035158 +0000 UTC m=+1796.232347253" Sep 08 02:28:39.468571 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:39.468539 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:28:39.470919 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:39.470896 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:28:46.042459 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:46.042427 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:53.249042 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.249006 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:53.249541 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.249209 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5694fc4746-8jbl4" podUID="07c62c84-4899-4e91-b3d2-b97263197dcc" containerName="manager" containerID="cri-o://ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce" gracePeriod=30 Sep 08 02:28:53.506483 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.506422 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:53.549916 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.549884 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs\") pod \"07c62c84-4899-4e91-b3d2-b97263197dcc\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " Sep 08 02:28:53.550105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.549933 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert\") pod \"07c62c84-4899-4e91-b3d2-b97263197dcc\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " Sep 08 02:28:53.550105 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.549993 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2vvg\" (UniqueName: \"kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg\") pod \"07c62c84-4899-4e91-b3d2-b97263197dcc\" (UID: \"07c62c84-4899-4e91-b3d2-b97263197dcc\") " Sep 08 02:28:53.552136 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.552104 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg" (OuterVolumeSpecName: "kube-api-access-t2vvg") pod "07c62c84-4899-4e91-b3d2-b97263197dcc" (UID: "07c62c84-4899-4e91-b3d2-b97263197dcc"). InnerVolumeSpecName "kube-api-access-t2vvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:28:53.552497 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.552463 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07c62c84-4899-4e91-b3d2-b97263197dcc" (UID: "07c62c84-4899-4e91-b3d2-b97263197dcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:28:53.552589 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.552472 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "07c62c84-4899-4e91-b3d2-b97263197dcc" (UID: "07c62c84-4899-4e91-b3d2-b97263197dcc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:28:53.650913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.650877 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-webhook-cert\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:28:53.650913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.650908 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2vvg\" (UniqueName: \"kubernetes.io/projected/07c62c84-4899-4e91-b3d2-b97263197dcc-kube-api-access-t2vvg\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:28:53.650913 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:53.650923 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c62c84-4899-4e91-b3d2-b97263197dcc-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:28:54.144924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.144887 2559 generic.go:356] "Generic (PLEG): container finished" podID="07c62c84-4899-4e91-b3d2-b97263197dcc" containerID="ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce" exitCode=0 Sep 08 02:28:54.145112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.144932 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-8jbl4" Sep 08 02:28:54.145112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.144933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-8jbl4" event={"ID":"07c62c84-4899-4e91-b3d2-b97263197dcc","Type":"ContainerDied","Data":"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce"} Sep 08 02:28:54.145112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.145039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-8jbl4" event={"ID":"07c62c84-4899-4e91-b3d2-b97263197dcc","Type":"ContainerDied","Data":"9f493d7df136deeaac64bab67eeb15dedd24f6fabc049d9f2607a030e20de6fc"} Sep 08 02:28:54.145112 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.145068 2559 scope.go:117] "RemoveContainer" containerID="ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce" Sep 08 02:28:54.155025 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.155002 2559 scope.go:117] "RemoveContainer" containerID="ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce" Sep 08 02:28:54.155347 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:28:54.155323 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce\": container with ID starting with ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce not found: ID does not exist" containerID="ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce" Sep 08 02:28:54.155402 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.155359 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce"} err="failed to get container status \"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce\": rpc error: code = NotFound desc = could not find container \"ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce\": container with ID starting with ff72fd2ee4ee88de47c66ad487d8fe60ee5f3cc121353308883e5021d19053ce not found: ID does not exist" Sep 08 02:28:54.171114 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.171081 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:54.175166 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:54.175142 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5694fc4746-8jbl4"] Sep 08 02:28:55.161101 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.161054 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5694fc4746-jwfp5"] Sep 08 02:28:55.161503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.161456 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c62c84-4899-4e91-b3d2-b97263197dcc" containerName="manager" Sep 08 02:28:55.161503 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.161467 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c62c84-4899-4e91-b3d2-b97263197dcc" containerName="manager" Sep 08 02:28:55.161578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.161537 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c62c84-4899-4e91-b3d2-b97263197dcc" containerName="manager" Sep 08 02:28:55.161578 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.161546 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7856e7b2-150e-4979-97f0-6fdcd2b3f1c2" containerName="istio-proxy" Sep 08 02:28:55.166007 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.165987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.170561 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.170540 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 02:28:55.170680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.170560 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 02:28:55.170680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.170569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc7lc\"" Sep 08 02:28:55.170680 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.170633 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 02:28:55.179177 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.179143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-jwfp5"] Sep 08 02:28:55.265213 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.265176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9c4\" (UniqueName: \"kubernetes.io/projected/3749653e-734a-4501-8c04-3a81db3403c0-kube-api-access-hc9c4\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.265456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.265227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-webhook-cert\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.265456 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.265252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-metrics-certs\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.319908 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.319871 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c62c84-4899-4e91-b3d2-b97263197dcc" path="/var/lib/kubelet/pods/07c62c84-4899-4e91-b3d2-b97263197dcc/volumes" Sep 08 02:28:55.365845 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.365808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9c4\" (UniqueName: \"kubernetes.io/projected/3749653e-734a-4501-8c04-3a81db3403c0-kube-api-access-hc9c4\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.366036 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.365873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-webhook-cert\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.366036 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.365908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-metrics-certs\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.368514 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.368491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-metrics-certs\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.368514 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.368500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3749653e-734a-4501-8c04-3a81db3403c0-webhook-cert\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.375817 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.375792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9c4\" (UniqueName: \"kubernetes.io/projected/3749653e-734a-4501-8c04-3a81db3403c0-kube-api-access-hc9c4\") pod \"maas-controller-5694fc4746-jwfp5\" (UID: \"3749653e-734a-4501-8c04-3a81db3403c0\") " pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.477645 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.477565 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:55.766442 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:55.766416 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5694fc4746-jwfp5"] Sep 08 02:28:55.769185 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:28:55.769152 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3749653e_734a_4501_8c04_3a81db3403c0.slice/crio-544b3d1e66dde894fb7e563a83eeb8ca28f1188eca2bec6b3060878f5905a6e3 WatchSource:0}: Error finding container 544b3d1e66dde894fb7e563a83eeb8ca28f1188eca2bec6b3060878f5905a6e3: Status 404 returned error can't find the container with id 544b3d1e66dde894fb7e563a83eeb8ca28f1188eca2bec6b3060878f5905a6e3 Sep 08 02:28:56.158134 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:56.158095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-jwfp5" event={"ID":"3749653e-734a-4501-8c04-3a81db3403c0","Type":"ContainerStarted","Data":"544b3d1e66dde894fb7e563a83eeb8ca28f1188eca2bec6b3060878f5905a6e3"} Sep 08 02:28:57.166233 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:57.166196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5694fc4746-jwfp5" event={"ID":"3749653e-734a-4501-8c04-3a81db3403c0","Type":"ContainerStarted","Data":"ffbfa8af609f453a38ee03ab126833be41d2be0f69cd1e88a9f5ecef043090be"} Sep 08 02:28:57.166738 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:57.166339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:28:57.185917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:28:57.185866 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5694fc4746-jwfp5" podStartSLOduration=1.7662626000000001 podStartE2EDuration="2.185850275s" totalImagesPullingTime="419.587675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:28:57.184632349 +0000 UTC m=+1818.361944424" watchObservedRunningTime="2026-09-08 02:28:57.185850275 +0000 UTC m=+1818.363162375" Sep 08 02:29:08.173864 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:29:08.173825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5694fc4746-jwfp5" Sep 08 02:30:05.993054 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:05.992968 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:30:05.993525 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:05.993197 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" podUID="cdd85030-540f-471c-a672-24d0d1b8fdd0" containerName="manager" containerID="cri-o://93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa" gracePeriod=10 Sep 08 02:30:07.351317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.351265 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:30:07.496725 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.496632 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qss85\" (UniqueName: \"kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85\") pod \"cdd85030-540f-471c-a672-24d0d1b8fdd0\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " Sep 08 02:30:07.496906 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.496735 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume\") pod \"cdd85030-540f-471c-a672-24d0d1b8fdd0\" (UID: \"cdd85030-540f-471c-a672-24d0d1b8fdd0\") " Sep 08 02:30:07.497106 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.497082 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cdd85030-540f-471c-a672-24d0d1b8fdd0" (UID: "cdd85030-540f-471c-a672-24d0d1b8fdd0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 02:30:07.498761 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.498733 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85" (OuterVolumeSpecName: "kube-api-access-qss85") pod "cdd85030-540f-471c-a672-24d0d1b8fdd0" (UID: "cdd85030-540f-471c-a672-24d0d1b8fdd0"). InnerVolumeSpecName "kube-api-access-qss85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:30:07.576574 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.576538 2559 generic.go:356] "Generic (PLEG): container finished" podID="cdd85030-540f-471c-a672-24d0d1b8fdd0" containerID="93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa" exitCode=0 Sep 08 02:30:07.576773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.576589 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" Sep 08 02:30:07.576773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.576622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" event={"ID":"cdd85030-540f-471c-a672-24d0d1b8fdd0","Type":"ContainerDied","Data":"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa"} Sep 08 02:30:07.576773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.576675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn" event={"ID":"cdd85030-540f-471c-a672-24d0d1b8fdd0","Type":"ContainerDied","Data":"2e51bbc31d7db8e4e7dc90934083a3e3f97d7a859ff60877f4f1121857664902"} Sep 08 02:30:07.576773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.576698 2559 scope.go:117] "RemoveContainer" containerID="93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa" Sep 08 02:30:07.587149 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.587129 2559 scope.go:117] "RemoveContainer" containerID="93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa" Sep 08 02:30:07.587432 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:30:07.587412 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa\": container with ID starting with 93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa not found: ID does not exist" containerID="93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa" Sep 08 02:30:07.587508 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.587446 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa"} err="failed to get container status \"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa\": rpc error: code = NotFound desc = could not find container \"93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa\": container with ID starting with 93f2db044189c186232100906f62d38a2541db3b62c0d10beee73239648257fa not found: ID does not exist" Sep 08 02:30:07.598216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.598186 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qss85\" (UniqueName: \"kubernetes.io/projected/cdd85030-540f-471c-a672-24d0d1b8fdd0-kube-api-access-qss85\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:30:07.598216 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.598219 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdd85030-540f-471c-a672-24d0d1b8fdd0-extensions-socket-volume\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:30:07.605171 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.605147 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:30:07.610911 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:07.610886 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-n9crn"] Sep 08 02:30:09.320674 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:30:09.320639 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd85030-540f-471c-a672-24d0d1b8fdd0" path="/var/lib/kubelet/pods/cdd85030-540f-471c-a672-24d0d1b8fdd0/volumes" Sep 08 02:31:06.651386 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.651349 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m"] Sep 08 02:31:06.651867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.651727 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd85030-540f-471c-a672-24d0d1b8fdd0" containerName="manager" Sep 08 02:31:06.651867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.651739 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd85030-540f-471c-a672-24d0d1b8fdd0" containerName="manager" Sep 08 02:31:06.651867 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.651803 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd85030-540f-471c-a672-24d0d1b8fdd0" containerName="manager" Sep 08 02:31:06.654845 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.654829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.657460 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.657441 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mc6hv\"" Sep 08 02:31:06.667096 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.667076 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m"] Sep 08 02:31:06.791610 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.791580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.791743 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.791626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbh4j\" (UniqueName: \"kubernetes.io/projected/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-kube-api-access-fbh4j\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.892071 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.892042 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbh4j\" (UniqueName: \"kubernetes.io/projected/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-kube-api-access-fbh4j\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.892185 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.892113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.892444 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.892426 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.901819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.901772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbh4j\" (UniqueName: \"kubernetes.io/projected/a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6-kube-api-access-fbh4j\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-qsh7m\" (UID: \"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:06.966152 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:06.966123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:07.254989 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:07.254961 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m"] Sep 08 02:31:07.257131 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:31:07.257106 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13ab4b1_b5ec_48a9_8d02_40af9e16cbf6.slice/crio-907d48a012a107b662f250cbbd5f410eab1597c9dfc8ea26a2ca428e06d16f18 WatchSource:0}: Error finding container 907d48a012a107b662f250cbbd5f410eab1597c9dfc8ea26a2ca428e06d16f18: Status 404 returned error can't find the container with id 907d48a012a107b662f250cbbd5f410eab1597c9dfc8ea26a2ca428e06d16f18 Sep 08 02:31:07.916117 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:07.916086 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" event={"ID":"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6","Type":"ContainerStarted","Data":"47d09066faacc1cd79a3dda1fd8045a2446d97fbf1bf8517110d4a97c1c40f46"} Sep 08 02:31:07.916562 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:07.916138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" event={"ID":"a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6","Type":"ContainerStarted","Data":"907d48a012a107b662f250cbbd5f410eab1597c9dfc8ea26a2ca428e06d16f18"} Sep 08 02:31:07.916562 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:07.916180 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:31:07.939732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:07.939683 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" podStartSLOduration=1.939666115 podStartE2EDuration="1.939666115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:31:07.93812507 +0000 UTC m=+1949.115437139" watchObservedRunningTime="2026-09-08 02:31:07.939666115 +0000 UTC m=+1949.116978186" Sep 08 02:31:18.925131 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:31:18.925098 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-qsh7m" Sep 08 02:32:37.554601 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.554570 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2"] Sep 08 02:32:37.558469 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.558445 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.569320 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.569271 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2"] Sep 08 02:32:37.592732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.592702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wch\" (UniqueName: \"kubernetes.io/projected/14f40472-1e97-4abf-ac06-62bfe95533e1-kube-api-access-76wch\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.592879 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.592759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-metrics-certs\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.592942 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.592878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f40472-1e97-4abf-ac06-62bfe95533e1-openshift-service-ca\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.592984 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.592970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-maas-api-tls\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.694226 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.694194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-maas-api-tls\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.694433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.694233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wch\" (UniqueName: \"kubernetes.io/projected/14f40472-1e97-4abf-ac06-62bfe95533e1-kube-api-access-76wch\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.694433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.694263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-metrics-certs\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.694433 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.694315 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f40472-1e97-4abf-ac06-62bfe95533e1-openshift-service-ca\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.694980 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.694957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f40472-1e97-4abf-ac06-62bfe95533e1-openshift-service-ca\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.696657 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.696626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-maas-api-tls\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.696751 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.696670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f40472-1e97-4abf-ac06-62bfe95533e1-metrics-certs\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.701775 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.701751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wch\" (UniqueName: \"kubernetes.io/projected/14f40472-1e97-4abf-ac06-62bfe95533e1-kube-api-access-76wch\") pod \"maas-api-cf5b5fd48-vgtg2\" (UID: \"14f40472-1e97-4abf-ac06-62bfe95533e1\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:37.875003 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:37.874908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:38.154771 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:38.154744 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2"] Sep 08 02:32:38.155843 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:32:38.155818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f40472_1e97_4abf_ac06_62bfe95533e1.slice/crio-e796f78ac4ae53175d5b4546c9b655af7b9f4f422174e17fdd28812d5dd74e19 WatchSource:0}: Error finding container e796f78ac4ae53175d5b4546c9b655af7b9f4f422174e17fdd28812d5dd74e19: Status 404 returned error can't find the container with id e796f78ac4ae53175d5b4546c9b655af7b9f4f422174e17fdd28812d5dd74e19 Sep 08 02:32:38.439866 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:38.439783 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" event={"ID":"14f40472-1e97-4abf-ac06-62bfe95533e1","Type":"ContainerStarted","Data":"e796f78ac4ae53175d5b4546c9b655af7b9f4f422174e17fdd28812d5dd74e19"} Sep 08 02:32:39.448878 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:39.448841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" event={"ID":"14f40472-1e97-4abf-ac06-62bfe95533e1","Type":"ContainerStarted","Data":"9901b2c1d1d3b676b193ad17e0ba443f97f63587e6fbe4403c2af1741b92c288"} Sep 08 02:32:39.449736 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:39.449708 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:39.469772 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:39.469725 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" podStartSLOduration=1.8852045259999999 podStartE2EDuration="2.469712593s" totalImagesPullingTime="584.508067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:32:39.467260058 +0000 UTC m=+2040.644572142" watchObservedRunningTime="2026-09-08 02:32:39.469712593 +0000 UTC m=+2040.647024665" Sep 08 02:32:45.459243 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:45.459214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-vgtg2" Sep 08 02:32:54.067654 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:54.067621 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:32:54.068211 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:54.067849 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerName="maas-api" containerID="cri-o://335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc" gracePeriod=30 Sep 08 02:32:54.176138 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:54.176091 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": dial tcp 10.132.0.58:8443: connect: connection refused" Sep 08 02:32:55.321223 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.321198 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:32:55.456599 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.456568 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nqn4\" (UniqueName: \"kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4\") pod \"0778aebe-bf73-40c3-a937-2e5efe46384e\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " Sep 08 02:32:55.456746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.456647 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs\") pod \"0778aebe-bf73-40c3-a937-2e5efe46384e\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " Sep 08 02:32:55.456746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.456684 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls\") pod \"0778aebe-bf73-40c3-a937-2e5efe46384e\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " Sep 08 02:32:55.456746 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.456726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca\") pod \"0778aebe-bf73-40c3-a937-2e5efe46384e\" (UID: \"0778aebe-bf73-40c3-a937-2e5efe46384e\") " Sep 08 02:32:55.457174 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.457135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0778aebe-bf73-40c3-a937-2e5efe46384e" (UID: "0778aebe-bf73-40c3-a937-2e5efe46384e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 02:32:55.458712 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.458682 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0778aebe-bf73-40c3-a937-2e5efe46384e" (UID: "0778aebe-bf73-40c3-a937-2e5efe46384e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:32:55.458920 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.458894 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4" (OuterVolumeSpecName: "kube-api-access-2nqn4") pod "0778aebe-bf73-40c3-a937-2e5efe46384e" (UID: "0778aebe-bf73-40c3-a937-2e5efe46384e"). InnerVolumeSpecName "kube-api-access-2nqn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 02:32:55.459047 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.459023 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0778aebe-bf73-40c3-a937-2e5efe46384e" (UID: "0778aebe-bf73-40c3-a937-2e5efe46384e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 02:32:55.543050 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.543023 2559 generic.go:356] "Generic (PLEG): container finished" podID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerID="335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc" exitCode=0 Sep 08 02:32:55.543198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.543073 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" Sep 08 02:32:55.543198 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.543071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" event={"ID":"0778aebe-bf73-40c3-a937-2e5efe46384e","Type":"ContainerDied","Data":"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc"} Sep 08 02:32:55.543328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.543209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z" event={"ID":"0778aebe-bf73-40c3-a937-2e5efe46384e","Type":"ContainerDied","Data":"26643b12ae73f4ace6414176b728a542a3ace2e3d583bb7140d2c2b58e4e44ab"} Sep 08 02:32:55.543328 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.543237 2559 scope.go:117] "RemoveContainer" containerID="335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc" Sep 08 02:32:55.555085 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.555053 2559 scope.go:117] "RemoveContainer" containerID="335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc" Sep 08 02:32:55.555544 ip-10-0-132-162 kubenswrapper[2559]: E0908 02:32:55.555519 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc\": container with ID starting with 335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc not found: ID does not exist" containerID="335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc" Sep 08 02:32:55.555660 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.555554 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc"} err="failed to get container status \"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc\": rpc error: code = NotFound desc = could not find container \"335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc\": container with ID starting with 335fee52633563b75a821eb4e2c0249f24ccf53f7bc51318e5564fac830d1cfc not found: ID does not exist" Sep 08 02:32:55.558055 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.558035 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0778aebe-bf73-40c3-a937-2e5efe46384e-openshift-service-ca\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:32:55.558055 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.558056 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqn4\" (UniqueName: \"kubernetes.io/projected/0778aebe-bf73-40c3-a937-2e5efe46384e-kube-api-access-2nqn4\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:32:55.558168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.558065 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-metrics-certs\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:32:55.558168 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.558073 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0778aebe-bf73-40c3-a937-2e5efe46384e-maas-api-tls\") on node \"ip-10-0-132-162.ec2.internal\" DevicePath \"\"" Sep 08 02:32:55.566367 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.566347 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:32:55.570937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:55.570918 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-dwb8z"] Sep 08 02:32:57.319019 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:32:57.318988 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" path="/var/lib/kubelet/pods/0778aebe-bf73-40c3-a937-2e5efe46384e/volumes" Sep 08 02:33:10.378113 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:10.378034 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67df555867-b7kqc_3a9c58fd-2883-4de6-834c-1f5e499addd4/authorino/0.log" Sep 08 02:33:15.406572 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:15.406538 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cs8mr_f5e36c67-3234-40df-bc1d-4e1075a72ac5/manager/0.log" Sep 08 02:33:15.530355 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:15.530325 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8nd6r_e863c99e-9508-412b-8687-65b83172f716/manager/0.log" Sep 08 02:33:15.655853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:15.655819 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5694fc4746-jwfp5_3749653e-734a-4501-8c04-3a81db3403c0/manager/0.log" Sep 08 02:33:16.011259 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:16.011233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-dcbvk_577fd71c-6d67-4013-829a-055a9cddf509/copy-manifests/0.log" Sep 08 02:33:16.017988 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:16.017959 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-dcbvk_577fd71c-6d67-4013-829a-055a9cddf509/manager/0.log" Sep 08 02:33:16.697817 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:16.697778 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cf5b5fd48-vgtg2_14f40472-1e97-4abf-ac06-62bfe95533e1/maas-api/0.log" Sep 08 02:33:17.011696 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:17.011616 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-mc985_b921561f-5241-4845-b6f7-7c7d66593fb8/postgres/0.log" Sep 08 02:33:17.761909 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:17.761873 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67df555867-b7kqc_3a9c58fd-2883-4de6-834c-1f5e499addd4/authorino/0.log" Sep 08 02:33:17.866773 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:17.866744 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zv6c5_daf6e428-2030-4b59-a818-3c3edfd23c75/manager/0.log" Sep 08 02:33:17.969585 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:17.969558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-gg6nq_77ebc803-9570-4ade-9655-2a0bb00d86cc/manager/0.log" Sep 08 02:33:18.072697 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:18.072630 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-hs74c_0f0ddaf3-bdbe-47db-b551-7cb364d74b2f/kuadrant-console-plugin/0.log" Sep 08 02:33:18.199493 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:18.199467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-qsh7m_a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6/manager/0.log" Sep 08 02:33:18.442872 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:18.442840 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wrfcd_1425544c-22ef-4323-9a0c-cf99361813fe/manager/0.log" Sep 08 02:33:18.995538 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:18.995509 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-9pnbr_edadf9cc-8d04-4944-ad81-0e29df02b766/kube-auth-proxy/0.log" Sep 08 02:33:19.442378 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:19.442345 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:33:19.553727 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:19.553685 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65ff7fbcfb-55zh8_041244a0-874b-4ed6-b34f-ad07dda18f5c/router/0.log" Sep 08 02:33:19.979925 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:19.979878 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-w9kbb_53246d60-6795-4095-8e78-d0411e6642ea/main/0.log" Sep 08 02:33:25.894510 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.894475 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86shj/must-gather-8lbm5"] Sep 08 02:33:25.895110 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.895074 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerName="maas-api" Sep 08 02:33:25.895110 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.895092 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerName="maas-api" Sep 08 02:33:25.895204 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.895175 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0778aebe-bf73-40c3-a937-2e5efe46384e" containerName="maas-api" Sep 08 02:33:25.898711 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.898691 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:25.901187 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.901163 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-86shj\"/\"default-dockercfg-5vlsl\"" Sep 08 02:33:25.901308 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.901163 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86shj\"/\"openshift-service-ca.crt\"" Sep 08 02:33:25.901750 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.901733 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86shj\"/\"kube-root-ca.crt\"" Sep 08 02:33:25.907689 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:25.907665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/must-gather-8lbm5"] Sep 08 02:33:26.019648 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.019621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-kube-api-access-nt8mj\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.019800 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.019678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-must-gather-output\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.120813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.120778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-kube-api-access-nt8mj\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.120972 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.120837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-must-gather-output\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.121197 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.121179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-must-gather-output\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.128804 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.128776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/255dbda5-02dc-40ce-bbf9-f591fa4d75f4-kube-api-access-nt8mj\") pod \"must-gather-8lbm5\" (UID: \"255dbda5-02dc-40ce-bbf9-f591fa4d75f4\") " pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.209964 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.209940 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/must-gather-8lbm5" Sep 08 02:33:26.489685 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.489660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/must-gather-8lbm5"] Sep 08 02:33:26.491385 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:33:26.491362 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod255dbda5_02dc_40ce_bbf9_f591fa4d75f4.slice/crio-8e1a9889383f052b35be9f33541bd6deebac218ede3b539089cb74b9767c38ab WatchSource:0}: Error finding container 8e1a9889383f052b35be9f33541bd6deebac218ede3b539089cb74b9767c38ab: Status 404 returned error can't find the container with id 8e1a9889383f052b35be9f33541bd6deebac218ede3b539089cb74b9767c38ab Sep 08 02:33:26.723662 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:26.723627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/must-gather-8lbm5" event={"ID":"255dbda5-02dc-40ce-bbf9-f591fa4d75f4","Type":"ContainerStarted","Data":"8e1a9889383f052b35be9f33541bd6deebac218ede3b539089cb74b9767c38ab"} Sep 08 02:33:27.740852 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:27.740814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/must-gather-8lbm5" event={"ID":"255dbda5-02dc-40ce-bbf9-f591fa4d75f4","Type":"ContainerStarted","Data":"8f5343f19dd2b873dc4da09e24e2bdaf2fbfb01b7caf1c8fa582174b89f73686"} Sep 08 02:33:27.741335 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:27.740861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/must-gather-8lbm5" event={"ID":"255dbda5-02dc-40ce-bbf9-f591fa4d75f4","Type":"ContainerStarted","Data":"b605100db9443a5e232c40dccb962af6f67ccfbfd23906050b8ed680e1dbc18d"} Sep 08 02:33:27.759176 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:27.759091 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86shj/must-gather-8lbm5" podStartSLOduration=1.898162443 podStartE2EDuration="2.759071909s" totalImagesPullingTime="860.909466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:33:27.755216484 +0000 UTC m=+2088.932528556" watchObservedRunningTime="2026-09-08 02:33:27.759071909 +0000 UTC m=+2088.936383983" Sep 08 02:33:28.935225 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:28.935188 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wpgqg_d7c3ff89-2fe9-404a-8b67-54e343ca26e6/global-pull-secret-syncer/0.log" Sep 08 02:33:29.021813 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:29.021780 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhfwz_69d8342c-24ae-4b85-87d9-bc9ce116e096/konnectivity-agent/0.log" Sep 08 02:33:29.101759 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:29.101730 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-162.ec2.internal_9d5df1337fbaf532327312e2b3b90e53/haproxy/0.log" Sep 08 02:33:33.203785 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.203745 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67df555867-b7kqc_3a9c58fd-2883-4de6-834c-1f5e499addd4/authorino/0.log" Sep 08 02:33:33.262405 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.262372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zv6c5_daf6e428-2030-4b59-a818-3c3edfd23c75/manager/0.log" Sep 08 02:33:33.300093 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.300060 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-gg6nq_77ebc803-9570-4ade-9655-2a0bb00d86cc/manager/0.log" Sep 08 02:33:33.346616 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.346588 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-hs74c_0f0ddaf3-bdbe-47db-b551-7cb364d74b2f/kuadrant-console-plugin/0.log" Sep 08 02:33:33.423865 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.423831 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-qsh7m_a13ab4b1-b5ec-48a9-8d02-40af9e16cbf6/manager/0.log" Sep 08 02:33:33.541886 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:33.541786 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wrfcd_1425544c-22ef-4323-9a0c-cf99361813fe/manager/0.log" Sep 08 02:33:35.375883 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:35.375840 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-dxsr9_9535a6aa-c1fe-4a1a-bcc1-ecabdfbb5bba/cluster-monitoring-operator/0.log" Sep 08 02:33:35.835356 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:35.835328 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvrsj_38ad3666-a60c-4480-99e5-1e8da4df2c5e/node-exporter/0.log" Sep 08 02:33:35.878505 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:35.878475 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvrsj_38ad3666-a60c-4480-99e5-1e8da4df2c5e/kube-rbac-proxy/0.log" Sep 08 02:33:35.921658 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:35.921629 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvrsj_38ad3666-a60c-4480-99e5-1e8da4df2c5e/init-textfile/0.log" Sep 08 02:33:36.436249 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:36.436218 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6d78946-662wg_0cf31d75-ff1f-40ea-8e9d-fbbe0180392a/telemeter-client/0.log" Sep 08 02:33:36.463411 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:36.463386 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6d78946-662wg_0cf31d75-ff1f-40ea-8e9d-fbbe0180392a/reload/0.log" Sep 08 02:33:36.488896 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:36.488857 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6d78946-662wg_0cf31d75-ff1f-40ea-8e9d-fbbe0180392a/kube-rbac-proxy/0.log" Sep 08 02:33:38.053550 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.053515 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-hjm2c_1bf8d592-ea4b-4ba0-83a0-503d3fcc27bc/networking-console-plugin/0.log" Sep 08 02:33:38.679066 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.679030 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:33:38.687048 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.687016 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/2.log" Sep 08 02:33:38.853317 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.852645 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d"] Sep 08 02:33:38.861148 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.861120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:38.865015 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.864972 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d"] Sep 08 02:33:38.961818 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.961708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-lib-modules\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:38.961818 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.961789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-podres\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:38.962093 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.961866 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd4m\" (UniqueName: \"kubernetes.io/projected/fb4ade16-856e-46e8-90ca-d568b3847ec2-kube-api-access-rjd4m\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:38.962093 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.961934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-sys\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:38.962093 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:38.962047 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-proc\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063380 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-proc\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-lib-modules\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-podres\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd4m\" (UniqueName: \"kubernetes.io/projected/fb4ade16-856e-46e8-90ca-d568b3847ec2-kube-api-access-rjd4m\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-sys\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-sys\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-proc\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-lib-modules\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.063853 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.063782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb4ade16-856e-46e8-90ca-d568b3847ec2-podres\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.076310 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.075857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd4m\" (UniqueName: \"kubernetes.io/projected/fb4ade16-856e-46e8-90ca-d568b3847ec2-kube-api-access-rjd4m\") pod \"perf-node-gather-daemonset-ztr5d\" (UID: \"fb4ade16-856e-46e8-90ca-d568b3847ec2\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.177169 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.176995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.270692 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.252309 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b7975f6cb-hnzfc_ad9a1638-4452-4cd0-80d9-83365238be0a/console/0.log" Sep 08 02:33:39.472498 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.472470 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d"] Sep 08 02:33:39.474139 ip-10-0-132-162 kubenswrapper[2559]: W0908 02:33:39.474113 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfb4ade16_856e_46e8_90ca_d568b3847ec2.slice/crio-76bc0f1bf6b6c557f935f51ba86414fe0bd80532cc82733d0c659ba1042d76c7 WatchSource:0}: Error finding container 76bc0f1bf6b6c557f935f51ba86414fe0bd80532cc82733d0c659ba1042d76c7: Status 404 returned error can't find the container with id 76bc0f1bf6b6c557f935f51ba86414fe0bd80532cc82733d0c659ba1042d76c7 Sep 08 02:33:39.526310 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.525867 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:33:39.538891 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.538710 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8j4cq_dacd11f3-44cb-4220-b7f6-ac652c06f334/console-operator/1.log" Sep 08 02:33:39.859275 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.859233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" event={"ID":"fb4ade16-856e-46e8-90ca-d568b3847ec2","Type":"ContainerStarted","Data":"861dd353b814ce5faa192febd0c19b68213e4c292cca591f8f7d71a2af19119c"} Sep 08 02:33:39.859476 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.859317 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" event={"ID":"fb4ade16-856e-46e8-90ca-d568b3847ec2","Type":"ContainerStarted","Data":"76bc0f1bf6b6c557f935f51ba86414fe0bd80532cc82733d0c659ba1042d76c7"} Sep 08 02:33:39.860166 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.860135 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:39.880191 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:39.880124 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" podStartSLOduration=1.8801006710000001 podStartE2EDuration="1.880100671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 02:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:33:39.876479303 +0000 UTC m=+2101.053791369" watchObservedRunningTime="2026-09-08 02:33:39.880100671 +0000 UTC m=+2101.057412744" Sep 08 02:33:40.005586 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:40.005552 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-pb7fl_2354d2ac-5552-45b8-8228-4fb9bbaf2eff/volume-data-source-validator/0.log" Sep 08 02:33:41.308859 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:41.308822 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xw95z_de250181-40c5-4498-9570-085aaaced35a/dns/0.log" Sep 08 02:33:41.336728 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:41.336692 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xw95z_de250181-40c5-4498-9570-085aaaced35a/kube-rbac-proxy/0.log" Sep 08 02:33:41.390524 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:41.390155 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5496l_082e2a56-3930-4c0a-99b7-910f08509976/dns-node-resolver/0.log" Sep 08 02:33:41.998842 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:41.998804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmcsm_128d5c0b-cee0-460b-a941-64eb1bd1c95a/node-ca/0.log" Sep 08 02:33:43.474013 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:43.473980 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-9pnbr_edadf9cc-8d04-4944-ad81-0e29df02b766/kube-auth-proxy/0.log" Sep 08 02:33:43.639412 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:43.639372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:33:43.662807 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:43.662780 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65ff7fbcfb-55zh8_041244a0-874b-4ed6-b34f-ad07dda18f5c/router/0.log" Sep 08 02:33:44.553732 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:44.553702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrx8x_3ee1da86-5399-4b71-9b2e-bcf1509942ed/serve-healthcheck-canary/0.log" Sep 08 02:33:45.036783 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:45.036751 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-q89gc_0e9f141c-f430-4cac-958d-841a7c761c57/insights-operator/0.log" Sep 08 02:33:45.037671 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:45.037644 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-q89gc_0e9f141c-f430-4cac-958d-841a7c761c57/insights-operator/1.log" Sep 08 02:33:45.062205 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:45.062175 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4vp99_160555fc-4c7d-45ff-b071-235080ad81a4/kube-rbac-proxy/0.log" Sep 08 02:33:45.089678 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:45.089654 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4vp99_160555fc-4c7d-45ff-b071-235080ad81a4/exporter/0.log" Sep 08 02:33:45.119878 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:45.119841 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4vp99_160555fc-4c7d-45ff-b071-235080ad81a4/extractor/0.log" Sep 08 02:33:46.888922 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:46.888892 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-ztr5d" Sep 08 02:33:47.620937 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:47.620906 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cs8mr_f5e36c67-3234-40df-bc1d-4e1075a72ac5/manager/0.log" Sep 08 02:33:47.883555 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:47.883525 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8nd6r_e863c99e-9508-412b-8687-65b83172f716/manager/0.log" Sep 08 02:33:48.103717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:48.103639 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5694fc4746-jwfp5_3749653e-734a-4501-8c04-3a81db3403c0/manager/0.log" Sep 08 02:33:48.317253 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:48.317221 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-dcbvk_577fd71c-6d67-4013-829a-055a9cddf509/manager/0.log" Sep 08 02:33:48.340717 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:48.340683 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-dcbvk_577fd71c-6d67-4013-829a-055a9cddf509/copy-manifests/0.log" Sep 08 02:33:49.897076 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:49.897037 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-9pnbr_edadf9cc-8d04-4944-ad81-0e29df02b766/kube-auth-proxy/0.log" Sep 08 02:33:50.037538 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:50.037503 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:33:50.054312 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:50.054261 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65ff7fbcfb-55zh8_041244a0-874b-4ed6-b34f-ad07dda18f5c/router/0.log" Sep 08 02:33:52.135073 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:52.134998 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-9pnbr_edadf9cc-8d04-4944-ad81-0e29df02b766/kube-auth-proxy/0.log" Sep 08 02:33:52.267924 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:52.267890 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-klkkd_df82d1d3-9c8e-4fc2-95c2-09655a1f5b41/payload-processing/0.log" Sep 08 02:33:52.291911 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:52.291875 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65ff7fbcfb-55zh8_041244a0-874b-4ed6-b34f-ad07dda18f5c/router/0.log" Sep 08 02:33:55.354917 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:55.354887 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-qbvhn_4ff518ec-91a9-4dcd-bd4c-62b4596db1ba/kube-storage-version-migrator-operator/1.log" Sep 08 02:33:55.356819 ip-10-0-132-162 kubenswrapper[2559]: I0908 02:33:55.356790 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-qbvhn_4ff518ec-91a9-4dcd-bd4c-62b4596db1ba/kube-storage-version-migrator-operator/0.log"