Sep 10 02:46:34.488055 ip-10-0-134-225 systemd[1]: Starting Kubernetes Kubelet... Sep 10 02:46:34.924669 ip-10-0-134-225 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:46:34.924669 ip-10-0-134-225 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 02:46:34.924669 ip-10-0-134-225 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:46:34.924669 ip-10-0-134-225 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 02:46:34.924669 ip-10-0-134-225 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:46:34.926809 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.926702 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 02:46:34.930120 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930103 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.930120 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930120 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930126 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930129 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930132 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930135 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930139 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930141 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930144 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930146 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930149 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930152 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930154 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930157 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930160 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930162 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930165 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930168 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930170 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930173 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930175 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.930187 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930178 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930181 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930183 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930186 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930188 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930191 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930193 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930197 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930202 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930206 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930208 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930211 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930213 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930216 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930219 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930221 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930224 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930226 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930229 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930231 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.930660 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930233 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930236 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930238 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930241 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930243 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930246 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930249 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930251 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930255 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930258 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930260 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930263 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930266 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930269 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930274 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930277 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930279 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930282 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930285 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930288 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.931200 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930291 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930293 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930296 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930299 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930302 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930304 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930307 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930310 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930312 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930315 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930317 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930320 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930323 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930327 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930329 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930332 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930334 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930337 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930340 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.931690 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930343 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930345 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930347 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930350 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930352 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930355 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930358 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930361 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930364 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930367 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930371 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930376 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930379 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930381 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930384 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930387 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930390 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930392 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.930396 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.932164 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931173 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931180 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931183 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931186 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931188 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931191 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931194 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931196 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931199 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931202 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931205 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931208 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931211 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931214 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931217 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931219 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931222 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931225 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931228 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931230 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.932717 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931234 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931237 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931239 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931242 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931244 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931247 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931250 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931252 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931254 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931257 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931259 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931262 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931265 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931267 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931269 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931272 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931274 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931277 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931280 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931283 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.933225 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931285 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931287 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931290 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931293 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931296 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931299 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931302 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931306 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931308 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931311 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931314 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931316 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931319 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931322 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931324 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931327 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931329 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931332 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931334 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.933741 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931337 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931339 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931341 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931344 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931346 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931349 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931351 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931354 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931356 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931358 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931361 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931363 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931366 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931370 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931373 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931376 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931382 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931386 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931389 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.934214 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931392 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931396 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931399 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931401 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931405 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931408 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931411 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931414 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931417 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931420 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931423 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931425 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931457 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931461 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931464 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931467 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931470 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931472 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931475 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931478 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.934681 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931480 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931919 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931925 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931928 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931931 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931934 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931937 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931940 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931943 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931948 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931952 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931956 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931959 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931962 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931965 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931969 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931972 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931975 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931977 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931980 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.935169 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931983 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931985 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931989 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931992 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931996 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.931999 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932001 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932004 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932007 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932009 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932012 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932014 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932018 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932021 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932024 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932026 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932029 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932031 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932034 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.935665 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932036 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932039 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932043 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932045 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932048 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932050 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932053 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932056 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932060 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932062 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932065 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932073 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932076 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932078 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932082 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932084 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932087 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932090 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932092 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932094 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.936148 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932097 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932100 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932102 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932105 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932107 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932110 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932112 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932115 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932117 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932120 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932122 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932125 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932127 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932130 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932134 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932137 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932139 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932142 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932145 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932148 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.936678 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932151 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932154 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932156 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932159 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932161 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932164 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932166 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932169 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932172 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932174 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932177 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932180 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932182 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932185 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932187 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932190 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932192 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932195 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932197 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932200 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.937178 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932202 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932303 2583 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932342 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932348 2583 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932354 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932359 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932364 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932368 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932373 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932376 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932379 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932383 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932387 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932390 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932393 2583 flags.go:64] FLAG: --cgroup-root="" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932396 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932400 2583 flags.go:64] FLAG: --client-ca-file="" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932403 2583 flags.go:64] FLAG: --cloud-provider="external" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932405 2583 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932410 2583 flags.go:64] FLAG: --cluster-domain="" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932413 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932416 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932420 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932423 2583 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 02:46:34.937671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932427 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932430 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932433 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932436 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932439 2583 flags.go:64] FLAG: --contention-profiling="false" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932442 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932445 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932448 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932451 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932455 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932458 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932461 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932463 2583 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932466 2583 flags.go:64] FLAG: --enable-server="true" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932471 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932475 2583 flags.go:64] FLAG: --event-burst="100" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932478 2583 flags.go:64] FLAG: --event-qps="50" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932481 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932483 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932487 2583 flags.go:64] FLAG: --eviction-hard="" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932491 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932494 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932497 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932500 2583 flags.go:64] FLAG: --eviction-soft="" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932503 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 02:46:34.938300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932506 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932509 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932512 2583 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932515 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932518 2583 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932522 2583 flags.go:64] FLAG: --feature-gates="" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932525 2583 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932528 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932531 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932534 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932537 2583 flags.go:64] FLAG: --healthz-port="10248" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932540 2583 flags.go:64] FLAG: --help="false" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932543 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-134-225.ec2.internal" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932546 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932549 2583 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932552 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932556 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932559 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932562 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932565 2583 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932568 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932573 2583 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932576 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932580 2583 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 02:46:34.938942 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932582 2583 flags.go:64] FLAG: --kube-reserved="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932585 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932588 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932591 2583 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932594 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932597 2583 flags.go:64] FLAG: --lock-file="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932600 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932603 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932606 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932611 2583 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932614 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932617 2583 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932620 2583 flags.go:64] FLAG: --logging-format="text" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932624 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932628 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932631 2583 flags.go:64] FLAG: --manifest-url="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932634 2583 flags.go:64] FLAG: --manifest-url-header="" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932638 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932641 2583 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932645 2583 flags.go:64] FLAG: --max-pods="110" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932648 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932651 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932654 2583 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932657 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932660 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 02:46:34.939533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932663 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932666 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932673 2583 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932676 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932681 2583 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932684 2583 flags.go:64] FLAG: --pod-cidr="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932687 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932693 2583 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932696 2583 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932699 2583 flags.go:64] FLAG: --pods-per-core="0" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932702 2583 flags.go:64] FLAG: --port="10250" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932705 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932708 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-016d2ecc9751ba2a7" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932711 2583 flags.go:64] FLAG: --qos-reserved="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932714 2583 flags.go:64] FLAG: --read-only-port="10255" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932717 2583 flags.go:64] FLAG: --register-node="true" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932732 2583 flags.go:64] FLAG: --register-with-taints="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932736 2583 flags.go:64] FLAG: --registry-burst="10" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932740 2583 flags.go:64] FLAG: --registry-qps="5" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932743 2583 flags.go:64] FLAG: --reserved-cpus="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932746 2583 flags.go:64] FLAG: --reserved-memory="" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932750 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932753 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932756 2583 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932759 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 02:46:34.940162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932762 2583 flags.go:64] FLAG: --runonce="false" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932765 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932768 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932771 2583 flags.go:64] FLAG: --seccomp-default="false" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932774 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932776 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932779 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932782 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932786 2583 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932788 2583 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932791 2583 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932795 2583 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932797 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932801 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932804 2583 flags.go:64] FLAG: --system-cgroups="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932807 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932812 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932815 2583 flags.go:64] FLAG: --tls-cert-file="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932820 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932824 2583 flags.go:64] FLAG: --tls-min-version="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932827 2583 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932830 2583 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932833 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932836 2583 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932839 2583 flags.go:64] FLAG: --v="2" Sep 10 02:46:34.940863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932843 2583 flags.go:64] FLAG: --version="false" Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932847 2583 flags.go:64] FLAG: --vmodule="" Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932851 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.932854 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932972 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932975 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932978 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932981 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932984 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932987 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932990 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932993 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932997 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.932999 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933002 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933005 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933007 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933010 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933013 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933015 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933018 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.941472 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933020 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933023 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933025 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933028 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933032 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933036 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933040 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933043 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933047 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933050 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933053 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933056 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933059 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933062 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933065 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933068 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933071 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933074 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933077 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.942230 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933080 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933083 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933086 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933088 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933091 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933094 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933097 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933099 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933102 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933104 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933107 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933109 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933112 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933114 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933117 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933119 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933122 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933126 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933129 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.942861 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933131 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933134 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933136 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933139 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933143 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933146 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933149 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933151 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933154 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933156 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933159 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933162 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933164 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933167 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933170 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933172 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933175 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933177 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933180 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.943342 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933183 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933186 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933190 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933193 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933196 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933198 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933201 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933204 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933206 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933209 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933212 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933216 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933219 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933221 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933224 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933227 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933229 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933232 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933236 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.944025 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933239 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933242 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933244 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933247 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933250 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.933252 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.933962 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.941653 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.941671 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.943415 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.943430 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943757 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943767 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943770 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943773 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.944491 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943777 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943780 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943783 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943785 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943788 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943791 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943794 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943797 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943800 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943803 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943806 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943809 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943811 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943814 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943817 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943819 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943821 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943824 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943827 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943830 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.944877 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943833 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943835 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943838 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943840 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943843 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943846 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943857 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943860 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943862 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943865 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943868 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943870 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943873 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943876 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943878 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943881 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943884 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943886 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943889 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943891 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.945377 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943894 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943896 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943899 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943902 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943904 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943907 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943909 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943912 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943914 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943917 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943919 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943922 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943924 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943927 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943930 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943933 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943935 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943939 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943942 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943944 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.945981 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943947 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943950 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943952 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943955 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943958 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943960 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943964 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943966 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943969 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943972 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943976 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943980 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943983 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943987 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943991 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943994 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.943998 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944000 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944003 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.946462 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944006 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944010 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944013 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944016 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944019 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944022 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944025 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944028 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944031 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944034 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944037 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944040 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944043 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944045 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944048 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.946945 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944051 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.944056 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944182 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944186 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944190 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944192 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944195 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944198 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944201 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944203 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944206 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944208 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944211 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944213 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944216 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:46:34.947363 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944219 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944222 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944224 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944227 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944229 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944240 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944243 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944245 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944248 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944250 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944253 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944256 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944259 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944262 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944265 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944267 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944270 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944272 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944275 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944278 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:46:34.947745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944280 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944283 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944285 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944289 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944292 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944295 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944298 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944300 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944303 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944305 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944308 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944312 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944315 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944317 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944320 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944323 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944326 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944330 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944332 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944335 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:46:34.948361 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944338 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944341 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944343 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944346 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944349 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944352 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944355 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944358 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944360 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944362 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944365 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944367 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944370 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944372 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944375 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944378 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944380 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944383 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944385 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:46:34.948872 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944388 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944390 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944393 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944395 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944398 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944400 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944403 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944406 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944408 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944411 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944413 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944416 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944418 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944421 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944424 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944426 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944428 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944432 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944435 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944437 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:46:34.949345 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944440 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944442 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944444 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944447 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944451 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944454 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:34.944457 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.944462 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.944586 2583 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.947016 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.948072 2583 server.go:1019] "Starting client certificate rotation" Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.948189 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 02:46:34.949855 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.948240 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 02:46:34.950990 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.950976 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 02:46:34.958021 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.957997 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 02:46:34.969452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.969432 2583 log.go:25] "Validated CRI v1 runtime API" Sep 10 02:46:34.972842 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.972826 2583 log.go:25] "Validated CRI v1 image API" Sep 10 02:46:34.973863 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.973847 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 02:46:34.978856 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.978831 2583 fs.go:135] Filesystem UUIDs: map[4d839acb-35a8-4c8b-a671-9d5aad394330:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ca1d26c2-750d-4a7e-9043-1a0e82d2d60e:/dev/nvme0n1p4] Sep 10 02:46:34.978924 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.978855 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 02:46:34.979292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.979270 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 02:46:34.984464 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.984341 2583 manager.go:216] Machine: {Timestamp:2026-09-10 02:46:34.983831075 +0000 UTC m=+0.379821570 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099871 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec240333923a767b887994baea964ecf SystemUUID:ec240333-923a-767b-8879-94baea964ecf BootID:690c3da1-9715-4527-a654-8d2c8bdf643d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:94:78:fb:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:94:78:fb:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:55:34:ad:76:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 02:46:34.984464 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.984454 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 02:46:34.984592 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.984568 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 02:46:34.988544 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.988517 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 02:46:34.988686 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.988548 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 02:46:34.989393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.989382 2583 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 02:46:34.989431 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.989395 2583 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 02:46:34.989431 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.989405 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 02:46:34.989431 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.989420 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 02:46:34.991307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.991294 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 02:46:34.996934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.996915 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 02:46:34.997075 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997064 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 02:46:34.997131 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997123 2583 kubelet.go:486] "Attempting to sync node with API server" Sep 10 02:46:34.997160 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997136 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 02:46:34.997160 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997154 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 02:46:34.997219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997164 2583 kubelet.go:398] "Adding apiserver pod source" Sep 10 02:46:34.997219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.997181 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 02:46:34.998964 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.998895 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 02:46:34.999092 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:34.998996 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 02:46:35.001949 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.001927 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 02:46:35.004892 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.004876 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 02:46:35.004958 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.004907 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 02:46:35.005085 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005075 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 02:46:35.005116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005090 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 02:46:35.005116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005097 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 02:46:35.005116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005102 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 02:46:35.005116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005110 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 02:46:35.005116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005116 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 02:46:35.005790 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005781 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 02:46:35.005790 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005790 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 02:46:35.005854 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005798 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 02:46:35.005854 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005805 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 02:46:35.005854 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005814 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 02:46:35.005854 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.005823 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 02:46:35.006738 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.006717 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 02:46:35.006782 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.006741 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 02:46:35.010653 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.010640 2583 server.go:1268] "Started kubelet" Sep 10 02:46:35.010803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.010778 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 02:46:35.010870 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.010773 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 02:46:35.010870 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.010848 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 02:46:35.012269 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.012251 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 02:46:35.012344 ip-10-0-134-225 systemd[1]: Started Kubernetes Kubelet. Sep 10 02:46:35.015148 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.015129 2583 server.go:310] "Adding debug handlers to kubelet server" Sep 10 02:46:35.016812 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.016694 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 02:46:35.016812 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.016717 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 02:46:35.016980 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.016825 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 02:46:35.016980 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.016844 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 02:46:35.017836 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.016668 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-225.ec2.internal.18d3d5a81ccc8293 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-225.ec2.internal,UID:ip-10-0-134-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-225.ec2.internal,},FirstTimestamp:2026-09-10 02:46:35.010605715 +0000 UTC m=+0.406596415,LastTimestamp:2026-09-10 02:46:35.010605715 +0000 UTC m=+0.406596415,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-225.ec2.internal,}" Sep 10 02:46:35.019210 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.019184 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 02:46:35.019878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.019855 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 02:46:35.019966 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.019936 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.020851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.020617 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 02:46:35.020851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.020648 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 02:46:35.020851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.020660 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 02:46:35.020851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.020814 2583 reconstruct.go:97] "Volume reconstruction finished" Sep 10 02:46:35.020851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.020822 2583 reconciler.go:29] "Reconciler: start to sync state" Sep 10 02:46:35.021967 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.021935 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 02:46:35.022171 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.022147 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 02:46:35.023238 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023220 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 02:46:35.023238 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023235 2583 factory.go:55] Registering systemd factory Sep 10 02:46:35.023238 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023241 2583 factory.go:223] Registration of the systemd container factory successfully Sep 10 02:46:35.023486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023474 2583 factory.go:153] Registering CRI-O factory Sep 10 02:46:35.023486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023488 2583 factory.go:223] Registration of the crio container factory successfully Sep 10 02:46:35.023561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023505 2583 factory.go:103] Registering Raw factory Sep 10 02:46:35.023561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023516 2583 manager.go:1196] Started watching for new ooms in manager Sep 10 02:46:35.023929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.023916 2583 manager.go:349] Starting recovery of all containers Sep 10 02:46:35.030017 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.029977 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jm2m" Sep 10 02:46:35.034480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.034457 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jm2m" Sep 10 02:46:35.037158 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.037140 2583 manager.go:354] Recovery completed Sep 10 02:46:35.041185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.041169 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.043052 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043032 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.043138 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043063 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.043138 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043076 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.043688 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043672 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 02:46:35.043688 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043686 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 02:46:35.043841 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043704 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 02:46:35.043841 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.043735 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.044563 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.044539 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.044645 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.044566 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.044645 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.044590 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.044645 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.044539 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-225.ec2.internal.18d3d5a81ebb8c4a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-225.ec2.internal,UID:ip-10-0-134-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-225.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-225.ec2.internal,},FirstTimestamp:2026-09-10 02:46:35.043048522 +0000 UTC m=+0.439039016,LastTimestamp:2026-09-10 02:46:35.043048522 +0000 UTC m=+0.439039016,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-225.ec2.internal,}" Sep 10 02:46:35.045911 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.045895 2583 policy_none.go:49] "None policy: Start" Sep 10 02:46:35.045994 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.045915 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 02:46:35.045994 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.045929 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 02:46:35.048129 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.048010 2583 policy_none.go:47] "Start" Sep 10 02:46:35.088427 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.088407 2583 manager.go:337] "Starting Device Plugin manager" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.088452 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.088465 2583 server.go:87] "Starting device plugin registration server" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.088897 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.088912 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.089009 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.089099 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.089108 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.089771 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 02:46:35.100477 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.089823 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.120121 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.120088 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 02:46:35.121239 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.121207 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 02:46:35.121353 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.121267 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 02:46:35.121353 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.121295 2583 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 02:46:35.121474 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.121395 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 02:46:35.123737 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.123708 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:35.189865 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.189749 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.191363 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.191343 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.191487 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.191377 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.191487 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.191388 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.191487 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.191409 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.198792 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.198773 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.198880 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.198798 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-225.ec2.internal\": node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.222171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.222131 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal"] Sep 10 02:46:35.222343 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.222222 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.224274 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.224256 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.224337 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.224290 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.224337 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.224300 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.224970 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.224955 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.226712 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.226700 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.226826 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.226811 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.226872 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.226840 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.227630 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227605 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.227699 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227635 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.227699 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227645 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.227699 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227605 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.227825 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227708 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.227825 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.227738 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.229925 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.229908 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.230019 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.229936 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:46:35.230708 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.230691 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:46:35.230813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.230745 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:46:35.230813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.230760 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:46:35.246330 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.246306 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-225.ec2.internal\" not found" node="ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.250928 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.250911 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-225.ec2.internal\" not found" node="ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.322417 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.322385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.322847 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.322830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.322903 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.322876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8c5ec7fd22f67956c3c0e7799a6de22-config\") pod \"kube-apiserver-proxy-ip-10-0-134-225.ec2.internal\" (UID: \"e8c5ec7fd22f67956c3c0e7799a6de22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.325446 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.325433 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.423927 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.423890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.423927 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.423928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8c5ec7fd22f67956c3c0e7799a6de22-config\") pod \"kube-apiserver-proxy-ip-10-0-134-225.ec2.internal\" (UID: \"e8c5ec7fd22f67956c3c0e7799a6de22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.424143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.423947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.424143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.423981 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.424143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.424001 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a619f86e883323954dd3877e7a62c9fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal\" (UID: \"a619f86e883323954dd3877e7a62c9fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.424143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.423986 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8c5ec7fd22f67956c3c0e7799a6de22-config\") pod \"kube-apiserver-proxy-ip-10-0-134-225.ec2.internal\" (UID: \"e8c5ec7fd22f67956c3c0e7799a6de22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.425986 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.425971 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.526834 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:35.526758 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:35.548944 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.548915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:35.554092 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:35.553717 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:37.499917 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.499891 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:37.600411 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.600358 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:37.701003 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.700918 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:37.743077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.743044 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:37.790654 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.790618 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:37.801054 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.801024 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-225.ec2.internal\" not found" Sep 10 02:46:37.821194 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.821169 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 02:46:37.821316 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.821298 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 02:46:37.821368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.821344 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 02:46:37.821405 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.821344 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 02:46:37.830779 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.830752 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:37.870541 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.870512 2583 apiserver.go:52] "Watching apiserver" Sep 10 02:46:37.877564 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.877540 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 02:46:37.878002 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.877979 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-bb84l","kube-system/konnectivity-agent-4m4vt","openshift-cluster-node-tuning-operator/tuned-zwpsf","openshift-image-registry/node-ca-zx4l6","openshift-multus/multus-2jwqs","openshift-multus/network-metrics-daemon-z98rn","openshift-network-diagnostics/network-check-target-dkxnf","openshift-network-operator/iptables-alerter-g2krg","openshift-ovn-kubernetes/ovnkube-node-d47m4","kube-system/global-pull-secret-syncer-7vbpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd","openshift-dns/node-resolver-vp6n2"] Sep 10 02:46:37.881448 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.881429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.883779 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.883760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:37.884020 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.883995 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 02:46:37.884020 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.884012 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 02:46:37.884162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.884041 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.884162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.884042 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 02:46:37.884162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.884064 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.884277 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.884177 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26p67\"" Sep 10 02:46:37.885712 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.885694 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7px2q\"" Sep 10 02:46:37.885812 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.885795 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 02:46:37.885905 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.885834 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 02:46:37.886487 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.886470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.888088 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.888069 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.888208 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.888193 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ltpr5\"" Sep 10 02:46:37.888208 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.888202 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.889217 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.889201 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:37.890889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.890848 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.891013 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.890996 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t6556\"" Sep 10 02:46:37.891077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.890998 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.891147 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.891130 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 02:46:37.891434 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.891417 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 02:46:37.892054 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.892031 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.892140 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.892115 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" Sep 10 02:46:37.894003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.893989 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 02:46:37.894003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.893994 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xpgld\"" Sep 10 02:46:37.894469 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.894450 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:37.894550 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.894524 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:37.897109 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.897089 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:37.897190 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.897169 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:37.899288 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.899398 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf7q\" (UniqueName: \"kubernetes.io/projected/2a0063e7-0b15-4581-b2ca-776e53c51b21-kube-api-access-xgf7q\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.899398 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-run\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899398 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899366 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-sys\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-host\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:37.899548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899451 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-modprobe-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899506 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-conf\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899538 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-serviceca\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-kubelet\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899609 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-hostroot\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-cnibin\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-systemd\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899800 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh5x\" (UniqueName: \"kubernetes.io/projected/0d496af9-8409-442d-bf47-a098ec37ec17-kube-api-access-kbh5x\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.899894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899846 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-socket-dir-parent\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.900155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899903 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-netns\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.900155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.899958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:37.900155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900000 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.900155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-var-lib-kubelet\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.900499 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-host\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.900499 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-tmp\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.900642 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5b4\" (UniqueName: \"kubernetes.io/projected/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-kube-api-access-dp5b4\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:37.900642 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900553 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 02:46:37.900813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:37.900889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edeb603a-a5ab-4613-9b62-0a59baee7e49-agent-certs\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:37.900946 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-etc-tuned\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.901001 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cnibin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901001 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.900987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-etc-kubernetes\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901073 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnp5\" (UniqueName: \"kubernetes.io/projected/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-kube-api-access-4fnp5\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901113 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-kube-api-access-4s9sp\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:37.901113 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901095 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.901113 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.901245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edeb603a-a5ab-4613-9b62-0a59baee7e49-konnectivity-ca\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:37.901245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysconfig\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.901245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-kubernetes\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.901245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901206 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-system-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-multus\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-conf-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-system-cni-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901310 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cni-binary-copy\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-k8s-cni-cncf-io\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901356 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-multus-certs\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-lib-modules\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-bin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.901498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.901460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-daemon-config\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:37.902583 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.902563 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.902661 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.902582 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c4x58\"" Sep 10 02:46:37.902661 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.902586 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.902804 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.902792 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 02:46:37.903474 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.903462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:37.905075 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.905060 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 02:46:37.905166 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.905128 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" Sep 10 02:46:37.905395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.905380 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 02:46:37.906122 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:37.906210 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:37.906178 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:37.906417 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906394 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rb9q9\"" Sep 10 02:46:37.906417 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906400 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 02:46:37.906561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906421 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 02:46:37.906561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906459 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 02:46:37.906561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906468 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.906561 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.906490 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.908894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.908878 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:37.910497 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.910462 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 02:41:36 +0000 UTC" deadline="2028-06-05 20:09:34.708306224 +0000 UTC" Sep 10 02:46:37.910497 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.910496 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15233h22m56.79781253s" Sep 10 02:46:37.911013 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.910997 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4djz\"" Sep 10 02:46:37.911101 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.910998 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.911399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.911379 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 02:46:37.911399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.911394 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal"] Sep 10 02:46:37.911497 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.911411 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.911497 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.911469 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:37.913928 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.913910 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal"] Sep 10 02:46:37.914322 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.914306 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 02:46:37.914390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.914379 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x4xgp\"" Sep 10 02:46:37.914431 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.914409 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 02:46:37.914594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.914578 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 02:46:37.924313 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.924289 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vwxqp" Sep 10 02:46:37.931526 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.931500 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vwxqp" Sep 10 02:46:37.995023 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:37.994535 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 02:46:38.002060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002038 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf7q\" (UniqueName: \"kubernetes.io/projected/2a0063e7-0b15-4581-b2ca-776e53c51b21-kube-api-access-xgf7q\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-host\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-kubelet\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-ovn\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002164 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-host\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-kubelet-config\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002264 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002305 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002321 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f135ece-4051-422d-a794-e4ac35d8c8b7-hosts-file\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-netns\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-systemd\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-socket-dir-parent\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ba51178-9185-4c38-ae2d-9f903bae3e0c-iptables-alerter-script\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-var-lib-kubelet\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.002434 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-systemd\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edeb603a-a5ab-4613-9b62-0a59baee7e49-agent-certs\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002487 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-socket-dir-parent\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-etc-kubernetes\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.002508 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:38.502486598 +0000 UTC m=+2.026343263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002515 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-var-lib-kubelet\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-etc-kubernetes\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002553 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-kube-api-access-4s9sp\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzs26\" (UniqueName: \"kubernetes.io/projected/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kube-api-access-wzs26\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-env-overrides\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.002727 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edeb603a-a5ab-4613-9b62-0a59baee7e49-konnectivity-ca\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002737 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-conf-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-script-lib\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002800 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-conf-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-k8s-cni-cncf-io\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002803 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-multus-certs\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-var-lib-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002896 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-k8s-cni-cncf-io\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-netd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002943 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-multus-certs\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.002995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-bin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-daemon-config\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.003606 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-etc-selinux\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-sys-fs\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003094 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dhd\" (UniqueName: \"kubernetes.io/projected/9f135ece-4051-422d-a794-e4ac35d8c8b7-kube-api-access-m8dhd\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-slash\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003139 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003162 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-run\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-sys\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003208 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-config\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0063e7-0b15-4581-b2ca-776e53c51b21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-bin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7h89\" (UniqueName: \"kubernetes.io/projected/aea63931-b129-4c99-9a59-f809d68496e3-kube-api-access-z7h89\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-dbus\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003300 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-run\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003306 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-modprobe-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003308 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edeb603a-a5ab-4613-9b62-0a59baee7e49-konnectivity-ca\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:38.004393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003323 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-sys\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-conf\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003413 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-modprobe-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003440 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-serviceca\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003487 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-d\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003517 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysctl-conf\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-kubelet\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-hostroot\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-socket-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003641 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-kubelet\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-cnibin\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh5x\" (UniqueName: \"kubernetes.io/projected/0d496af9-8409-442d-bf47-a098ec37ec17-kube-api-access-kbh5x\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003701 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-hostroot\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-netns\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003717 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-cnibin\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-registration-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003750 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-multus-daemon-config\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003787 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-run-netns\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003799 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-etc-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-bin\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea63931-b129-4c99-9a59-f809d68496e3-ovn-node-metrics-cert\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003897 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-serviceca\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-host\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-tmp\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.003941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5b4\" (UniqueName: \"kubernetes.io/projected/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-kube-api-access-dp5b4\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004000 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-host\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004009 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f135ece-4051-422d-a794-e4ac35d8c8b7-tmp-dir\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-systemd-units\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-log-socket\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-etc-tuned\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cnibin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnp5\" (UniqueName: \"kubernetes.io/projected/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-kube-api-access-4fnp5\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.005669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-device-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysconfig\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004223 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-sysconfig\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cnibin\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-kubernetes\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-system-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-multus\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ba51178-9185-4c38-ae2d-9f903bae3e0c-host-slash\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004416 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-system-cni-dir\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004416 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-etc-kubernetes\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6dl\" (UniqueName: \"kubernetes.io/projected/0ba51178-9185-4c38-ae2d-9f903bae3e0c-kube-api-access-7h6dl\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-host-var-lib-cni-multus\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-system-cni-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cni-binary-copy\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004515 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0063e7-0b15-4581-b2ca-776e53c51b21-system-cni-dir\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-node-log\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-lib-modules\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-systemd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.006722 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.004711 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d496af9-8409-442d-bf47-a098ec37ec17-lib-modules\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006722 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.005564 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-cni-binary-copy\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.006722 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.006365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-etc-tuned\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006722 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.006458 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d496af9-8409-442d-bf47-a098ec37ec17-tmp\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.006722 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.006698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edeb603a-a5ab-4613-9b62-0a59baee7e49-agent-certs\") pod \"konnectivity-agent-4m4vt\" (UID: \"edeb603a-a5ab-4613-9b62-0a59baee7e49\") " pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:38.020794 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.020769 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf7q\" (UniqueName: \"kubernetes.io/projected/2a0063e7-0b15-4581-b2ca-776e53c51b21-kube-api-access-xgf7q\") pod \"multus-additional-cni-plugins-bb84l\" (UID: \"2a0063e7-0b15-4581-b2ca-776e53c51b21\") " pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.020984 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.020968 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9sp\" (UniqueName: \"kubernetes.io/projected/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-kube-api-access-4s9sp\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:38.025778 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.025747 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5b4\" (UniqueName: \"kubernetes.io/projected/0ab99a0f-f0bc-4500-b53e-5b10a6cfba07-kube-api-access-dp5b4\") pod \"node-ca-zx4l6\" (UID: \"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07\") " pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.026180 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.026165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnp5\" (UniqueName: \"kubernetes.io/projected/ac949100-bd58-4a5f-90fd-4c2b55d72cf4-kube-api-access-4fnp5\") pod \"multus-2jwqs\" (UID: \"ac949100-bd58-4a5f-90fd-4c2b55d72cf4\") " pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.036157 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.036134 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh5x\" (UniqueName: \"kubernetes.io/projected/0d496af9-8409-442d-bf47-a098ec37ec17-kube-api-access-kbh5x\") pod \"tuned-zwpsf\" (UID: \"0d496af9-8409-442d-bf47-a098ec37ec17\") " pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.077344 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.077311 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c5ec7fd22f67956c3c0e7799a6de22.slice/crio-7d61aa972222216d91571472379208011ce961769673840e39751fe662a5aa2d WatchSource:0}: Error finding container 7d61aa972222216d91571472379208011ce961769673840e39751fe662a5aa2d: Status 404 returned error can't find the container with id 7d61aa972222216d91571472379208011ce961769673840e39751fe662a5aa2d Sep 10 02:46:38.077896 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.077874 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda619f86e883323954dd3877e7a62c9fd.slice/crio-8178dbc3f2f5abb0dc6fe93293edb5566cd0467818acc91c8e72ed1c07fa9a57 WatchSource:0}: Error finding container 8178dbc3f2f5abb0dc6fe93293edb5566cd0467818acc91c8e72ed1c07fa9a57: Status 404 returned error can't find the container with id 8178dbc3f2f5abb0dc6fe93293edb5566cd0467818acc91c8e72ed1c07fa9a57 Sep 10 02:46:38.083983 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.083968 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:46:38.104796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.104772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzs26\" (UniqueName: \"kubernetes.io/projected/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kube-api-access-wzs26\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.104934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.104800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-env-overrides\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.104934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.104819 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-script-lib\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105006 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.104944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:38.105006 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.104976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-var-lib-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105006 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-netd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105145 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-netd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105145 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105071 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-var-lib-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105145 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-etc-selinux\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-sys-fs\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dhd\" (UniqueName: \"kubernetes.io/projected/9f135ece-4051-422d-a794-e4ac35d8c8b7-kube-api-access-m8dhd\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-slash\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105233 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-etc-selinux\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105248 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105261 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-sys-fs\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-config\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105306 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7h89\" (UniqueName: \"kubernetes.io/projected/aea63931-b129-4c99-9a59-f809d68496e3-kube-api-access-z7h89\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105277 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-env-overrides\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105324 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-slash\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-dbus\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-socket-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105389 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-registration-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105405 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-dbus\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-etc-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105450 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-etc-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-bin\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea63931-b129-4c99-9a59-f809d68496e3-ovn-node-metrics-cert\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105504 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-socket-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f135ece-4051-422d-a794-e4ac35d8c8b7-tmp-dir\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105532 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-cni-bin\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105510 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-script-lib\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-registration-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.105633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-systemd-units\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-log-socket\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105689 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea63931-b129-4c99-9a59-f809d68496e3-ovnkube-config\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105715 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-systemd-units\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-device-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ba51178-9185-4c38-ae2d-9f903bae3e0c-host-slash\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105765 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-log-socket\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105764 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f135ece-4051-422d-a794-e4ac35d8c8b7-tmp-dir\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6dl\" (UniqueName: \"kubernetes.io/projected/0ba51178-9185-4c38-ae2d-9f903bae3e0c-kube-api-access-7h6dl\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105791 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-device-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ba51178-9185-4c38-ae2d-9f903bae3e0c-host-slash\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-node-log\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-systemd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105915 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-node-log\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-systemd\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.105982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-kubelet\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-ovn\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-ovn\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-kubelet-config\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-kubelet\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106106 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-kubelet-config\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f135ece-4051-422d-a794-e4ac35d8c8b7-hosts-file\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.106179 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-netns\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f135ece-4051-422d-a794-e4ac35d8c8b7-hosts-file\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ba51178-9185-4c38-ae2d-9f903bae3e0c-iptables-alerter-script\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.106235 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:46:38.606220414 +0000 UTC m=+2.130077100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-run-openvswitch\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea63931-b129-4c99-9a59-f809d68496e3-host-run-netns\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.106878 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.106570 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ba51178-9185-4c38-ae2d-9f903bae3e0c-iptables-alerter-script\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.107614 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.107598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea63931-b129-4c99-9a59-f809d68496e3-ovn-node-metrics-cert\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.116697 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.116678 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:38.116759 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.116700 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:38.116759 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.116711 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:38.116822 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.116767 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:38.616749942 +0000 UTC m=+2.140606607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:38.118980 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.118961 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dhd\" (UniqueName: \"kubernetes.io/projected/9f135ece-4051-422d-a794-e4ac35d8c8b7-kube-api-access-m8dhd\") pod \"node-resolver-vp6n2\" (UID: \"9f135ece-4051-422d-a794-e4ac35d8c8b7\") " pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.119100 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.119083 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7h89\" (UniqueName: \"kubernetes.io/projected/aea63931-b129-4c99-9a59-f809d68496e3-kube-api-access-z7h89\") pod \"ovnkube-node-d47m4\" (UID: \"aea63931-b129-4c99-9a59-f809d68496e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.119525 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.119509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzs26\" (UniqueName: \"kubernetes.io/projected/a27cb825-d5ee-496d-bf3a-5b0d4cad582f-kube-api-access-wzs26\") pod \"aws-ebs-csi-driver-node-97bdd\" (UID: \"a27cb825-d5ee-496d-bf3a-5b0d4cad582f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.119578 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.119544 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6dl\" (UniqueName: \"kubernetes.io/projected/0ba51178-9185-4c38-ae2d-9f903bae3e0c-kube-api-access-7h6dl\") pod \"iptables-alerter-g2krg\" (UID: \"0ba51178-9185-4c38-ae2d-9f903bae3e0c\") " pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.212075 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.212007 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bb84l" Sep 10 02:46:38.221963 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.221931 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0063e7_0b15_4581_b2ca_776e53c51b21.slice/crio-9477faec91ae37761cff1c70eac181234cbdc9bb0e2e40ba57beb4f182221fdc WatchSource:0}: Error finding container 9477faec91ae37761cff1c70eac181234cbdc9bb0e2e40ba57beb4f182221fdc: Status 404 returned error can't find the container with id 9477faec91ae37761cff1c70eac181234cbdc9bb0e2e40ba57beb4f182221fdc Sep 10 02:46:38.226013 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.225995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:46:38.233978 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.233950 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeb603a_a5ab_4613_9b62_0a59baee7e49.slice/crio-25903b6ce54f52e6bc2892b741e35cb8070be64fce25d604bb5e1989a991f0b1 WatchSource:0}: Error finding container 25903b6ce54f52e6bc2892b741e35cb8070be64fce25d604bb5e1989a991f0b1: Status 404 returned error can't find the container with id 25903b6ce54f52e6bc2892b741e35cb8070be64fce25d604bb5e1989a991f0b1 Sep 10 02:46:38.239552 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.239531 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" Sep 10 02:46:38.247294 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.247272 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d496af9_8409_442d_bf47_a098ec37ec17.slice/crio-e726e6d3a73b3f039ca0996508dcea62091289f7bd73a653b3063cb072e9980e WatchSource:0}: Error finding container e726e6d3a73b3f039ca0996508dcea62091289f7bd73a653b3063cb072e9980e: Status 404 returned error can't find the container with id e726e6d3a73b3f039ca0996508dcea62091289f7bd73a653b3063cb072e9980e Sep 10 02:46:38.255389 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.255371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zx4l6" Sep 10 02:46:38.262188 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.262169 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jwqs" Sep 10 02:46:38.262958 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.262938 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab99a0f_f0bc_4500_b53e_5b10a6cfba07.slice/crio-50106fd71d007138069eee3fbf46e463832e7bb7d4ad0f2d964e8953bb31e63d WatchSource:0}: Error finding container 50106fd71d007138069eee3fbf46e463832e7bb7d4ad0f2d964e8953bb31e63d: Status 404 returned error can't find the container with id 50106fd71d007138069eee3fbf46e463832e7bb7d4ad0f2d964e8953bb31e63d Sep 10 02:46:38.270151 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.270129 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac949100_bd58_4a5f_90fd_4c2b55d72cf4.slice/crio-2912fb6ab9e547ab7f0a2f31cac487cead05429f9f1ff8f8cde8b6c66a0972e1 WatchSource:0}: Error finding container 2912fb6ab9e547ab7f0a2f31cac487cead05429f9f1ff8f8cde8b6c66a0972e1: Status 404 returned error can't find the container with id 2912fb6ab9e547ab7f0a2f31cac487cead05429f9f1ff8f8cde8b6c66a0972e1 Sep 10 02:46:38.280295 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.280276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2krg" Sep 10 02:46:38.288302 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.288279 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba51178_9185_4c38_ae2d_9f903bae3e0c.slice/crio-e90426d611a66fb4fe0056c16a78223dba3e65a4e80b0fc1dc8529b7cd2eae63 WatchSource:0}: Error finding container e90426d611a66fb4fe0056c16a78223dba3e65a4e80b0fc1dc8529b7cd2eae63: Status 404 returned error can't find the container with id e90426d611a66fb4fe0056c16a78223dba3e65a4e80b0fc1dc8529b7cd2eae63 Sep 10 02:46:38.296483 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.296461 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:46:38.305527 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.305507 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea63931_b129_4c99_9a59_f809d68496e3.slice/crio-76651ac986b8dfc1a7750924ee712616a1463838ff97f41f2710fbeefe8cc4ee WatchSource:0}: Error finding container 76651ac986b8dfc1a7750924ee712616a1463838ff97f41f2710fbeefe8cc4ee: Status 404 returned error can't find the container with id 76651ac986b8dfc1a7750924ee712616a1463838ff97f41f2710fbeefe8cc4ee Sep 10 02:46:38.317246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.317228 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" Sep 10 02:46:38.325941 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.325917 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27cb825_d5ee_496d_bf3a_5b0d4cad582f.slice/crio-75e0ba0e282b6b72794d6f73cc82e03b1bcbb3bb8a8da279e0e81c569429707e WatchSource:0}: Error finding container 75e0ba0e282b6b72794d6f73cc82e03b1bcbb3bb8a8da279e0e81c569429707e: Status 404 returned error can't find the container with id 75e0ba0e282b6b72794d6f73cc82e03b1bcbb3bb8a8da279e0e81c569429707e Sep 10 02:46:38.326662 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.326641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp6n2" Sep 10 02:46:38.336072 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:46:38.336047 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f135ece_4051_422d_a794_e4ac35d8c8b7.slice/crio-1363eb04c89537e216cace4970461c6edc150e7583be1a954ecc0a4d12f7d5c6 WatchSource:0}: Error finding container 1363eb04c89537e216cace4970461c6edc150e7583be1a954ecc0a4d12f7d5c6: Status 404 returned error can't find the container with id 1363eb04c89537e216cace4970461c6edc150e7583be1a954ecc0a4d12f7d5c6 Sep 10 02:46:38.508063 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.507973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:38.508228 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.508114 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:38.508228 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.508181 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:39.508162173 +0000 UTC m=+3.032018854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:38.608505 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.608466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.608637 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.608607 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:38.608692 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.608681 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:46:39.608662529 +0000 UTC m=+3.132519201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:38.709694 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.709657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:38.709898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.709819 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:38.709898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.709839 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:38.709898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.709850 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:38.710072 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.709924 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:39.709906432 +0000 UTC m=+3.233763111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:38.939826 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.939664 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:37 +0000 UTC" deadline="2028-03-22 03:52:09.029380868 +0000 UTC" Sep 10 02:46:38.939826 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.939725 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13417h5m30.089659883s" Sep 10 02:46:38.995090 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:38.994583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:38.995090 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:38.994721 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:39.037767 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.037652 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp6n2" event={"ID":"9f135ece-4051-422d-a794-e4ac35d8c8b7","Type":"ContainerStarted","Data":"1363eb04c89537e216cace4970461c6edc150e7583be1a954ecc0a4d12f7d5c6"} Sep 10 02:46:39.040648 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.040559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jwqs" event={"ID":"ac949100-bd58-4a5f-90fd-4c2b55d72cf4","Type":"ContainerStarted","Data":"2912fb6ab9e547ab7f0a2f31cac487cead05429f9f1ff8f8cde8b6c66a0972e1"} Sep 10 02:46:39.053994 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.053953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zx4l6" event={"ID":"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07","Type":"ContainerStarted","Data":"50106fd71d007138069eee3fbf46e463832e7bb7d4ad0f2d964e8953bb31e63d"} Sep 10 02:46:39.060037 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.059999 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" event={"ID":"0d496af9-8409-442d-bf47-a098ec37ec17","Type":"ContainerStarted","Data":"e726e6d3a73b3f039ca0996508dcea62091289f7bd73a653b3063cb072e9980e"} Sep 10 02:46:39.091310 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.091270 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4m4vt" event={"ID":"edeb603a-a5ab-4613-9b62-0a59baee7e49","Type":"ContainerStarted","Data":"25903b6ce54f52e6bc2892b741e35cb8070be64fce25d604bb5e1989a991f0b1"} Sep 10 02:46:39.116195 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.116153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerStarted","Data":"9477faec91ae37761cff1c70eac181234cbdc9bb0e2e40ba57beb4f182221fdc"} Sep 10 02:46:39.140747 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.140701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" event={"ID":"a619f86e883323954dd3877e7a62c9fd","Type":"ContainerStarted","Data":"8178dbc3f2f5abb0dc6fe93293edb5566cd0467818acc91c8e72ed1c07fa9a57"} Sep 10 02:46:39.175028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.174985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" event={"ID":"a27cb825-d5ee-496d-bf3a-5b0d4cad582f","Type":"ContainerStarted","Data":"75e0ba0e282b6b72794d6f73cc82e03b1bcbb3bb8a8da279e0e81c569429707e"} Sep 10 02:46:39.210049 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.209956 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"76651ac986b8dfc1a7750924ee712616a1463838ff97f41f2710fbeefe8cc4ee"} Sep 10 02:46:39.233675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.233630 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2krg" event={"ID":"0ba51178-9185-4c38-ae2d-9f903bae3e0c","Type":"ContainerStarted","Data":"e90426d611a66fb4fe0056c16a78223dba3e65a4e80b0fc1dc8529b7cd2eae63"} Sep 10 02:46:39.247204 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.247166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" event={"ID":"e8c5ec7fd22f67956c3c0e7799a6de22","Type":"ContainerStarted","Data":"7d61aa972222216d91571472379208011ce961769673840e39751fe662a5aa2d"} Sep 10 02:46:39.410156 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.410023 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:39.516440 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.516006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:39.516440 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.516187 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:39.516440 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.516258 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:41.516237741 +0000 UTC m=+5.040094424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:39.617697 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.616947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:39.617697 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.617107 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:39.617697 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.617177 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:46:41.617158672 +0000 UTC m=+5.141015352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:39.718436 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.718397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:39.718723 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.718704 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:39.718786 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.718736 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:39.718786 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.718750 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:39.718938 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.718814 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:41.718795617 +0000 UTC m=+5.242652288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:39.940604 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.940486 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:37 +0000 UTC" deadline="2028-02-19 22:11:58.542347534 +0000 UTC" Sep 10 02:46:39.940604 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.940526 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12667h25m18.60182552s" Sep 10 02:46:39.995358 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.994504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:39.995358 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.994633 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:39.995358 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:39.994716 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:39.995358 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:39.994802 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:40.456436 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:40.456404 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:41.002609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.001940 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:41.002609 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.002119 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:41.021763 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.021731 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:46:41.531954 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.531681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:41.532117 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.532092 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:41.532188 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.532166 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.532146028 +0000 UTC m=+9.056002707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:41.633126 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.632628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:41.633126 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.632808 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:41.633126 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.632894 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.632874457 +0000 UTC m=+9.156731142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:41.733767 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.733080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:41.733767 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.733277 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:41.733767 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.733298 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:41.733767 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.733310 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:41.733767 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.733368 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.733349929 +0000 UTC m=+9.257206599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:41.994368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.994290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:41.994524 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.994434 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:41.994740 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:41.994702 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:41.994842 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:41.994818 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:42.994996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:42.994961 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:42.995467 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:42.995094 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:43.994652 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:43.994613 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:43.994652 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:43.994656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:43.994940 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:43.994751 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:43.994940 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:43.994831 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:44.994596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:44.994551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:44.995080 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:44.994687 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:45.564828 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:45.564789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:45.565025 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.564997 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:45.565088 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.565063 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:53.565049765 +0000 UTC m=+17.088906443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:45.665406 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:45.665364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:45.665572 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.665518 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:45.665628 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.665590 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:46:53.665571173 +0000 UTC m=+17.189427855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:45.765796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:45.765704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:45.765977 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.765901 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:45.765977 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.765925 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:45.765977 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.765937 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:45.766156 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.766008 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:53.765988956 +0000 UTC m=+17.289845635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:45.994142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:45.994112 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:45.994325 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.994245 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:45.994325 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:45.994301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:45.994443 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:45.994391 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:46.998448 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:46.997922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:46.998448 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:46.998051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:47.994463 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:47.994424 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:47.994664 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:47.994477 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:47.994664 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:47.994574 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:47.994765 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:47.994705 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:48.993975 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:48.993944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:48.994377 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:48.994053 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:49.994554 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:49.994506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:49.994554 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:49.994533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:49.995033 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:49.994657 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:49.995033 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:49.994765 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:50.997376 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:50.997345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:50.997810 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:50.997470 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:51.993696 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:51.993662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:51.993696 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:51.993698 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:51.994001 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:51.993793 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:51.994001 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:51.993899 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:52.993744 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:52.993702 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:52.994229 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:52.993848 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:53.627757 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:53.627723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:53.627962 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.627899 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:53.628025 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.627977 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:09.627955714 +0000 UTC m=+33.151812384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:46:53.728414 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:53.728382 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:53.728606 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.728521 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:53.728606 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.728582 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:47:09.728564331 +0000 UTC m=+33.252420998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:46:53.829239 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:53.829207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:53.829408 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.829377 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:46:53.829408 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.829398 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:46:53.829408 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.829409 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:53.829542 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.829470 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:09.829450727 +0000 UTC m=+33.353307416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:46:53.994451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:53.994363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:53.994451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:53.994412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:53.994946 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.994515 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:53.994946 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:53.994623 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:54.994301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:54.994260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:54.994612 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:54.994393 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:55.993928 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:55.993890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:55.993928 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:55.993920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:55.994151 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:55.994038 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:55.994228 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:55.994197 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:56.994352 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:56.994332 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:56.994730 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:56.994412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:57.289966 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.289663 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" event={"ID":"0d496af9-8409-442d-bf47-a098ec37ec17","Type":"ContainerStarted","Data":"131646863ff456bb77aea2407bc8a17015f7d201176b0d0511f44357683f62ac"} Sep 10 02:46:57.293389 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.293360 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" event={"ID":"e8c5ec7fd22f67956c3c0e7799a6de22","Type":"ContainerStarted","Data":"345650637c564c92e2219ebbc56c9bb97f532783f15c627acb9c422fe5da1d86"} Sep 10 02:46:57.296597 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.296481 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jwqs" event={"ID":"ac949100-bd58-4a5f-90fd-4c2b55d72cf4","Type":"ContainerStarted","Data":"1ae8e5dcd49832cbe0c079e09084a14ca88ad5c7c3a1426e4c29150bff56ed6f"} Sep 10 02:46:57.307667 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.307614 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwpsf" podStartSLOduration=1.583964613 podStartE2EDuration="20.307587345s" totalImagesPullingTime="18.723622732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:57.307176674 +0000 UTC m=+20.831033361" watchObservedRunningTime="2026-09-10 02:46:57.307587345 +0000 UTC m=+20.831444031" Sep 10 02:46:57.319751 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.319712 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-225.ec2.internal" podStartSLOduration=20.31969929 podStartE2EDuration="20.31969929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:57.319614443 +0000 UTC m=+20.843471131" watchObservedRunningTime="2026-09-10 02:46:57.31969929 +0000 UTC m=+20.843555977" Sep 10 02:46:57.347216 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.347164 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2jwqs" podStartSLOduration=1.6301793930000001 podStartE2EDuration="20.347142016s" totalImagesPullingTime="18.716962623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:57.347125702 +0000 UTC m=+20.870982414" watchObservedRunningTime="2026-09-10 02:46:57.347142016 +0000 UTC m=+20.870998707" Sep 10 02:46:57.994354 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.994201 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:57.994830 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:57.994271 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:57.994830 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:57.994431 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:46:57.994830 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:57.994540 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:58.300838 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.300785 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp6n2" event={"ID":"9f135ece-4051-422d-a794-e4ac35d8c8b7","Type":"ContainerStarted","Data":"ca87499db79ffd23ab990092b9b3298721703067a414c466093823d537238b22"} Sep 10 02:46:58.302812 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.302778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zx4l6" event={"ID":"0ab99a0f-f0bc-4500-b53e-5b10a6cfba07","Type":"ContainerStarted","Data":"a3ec39a66181bc5c9a3a7f93cac42b83dd19cf8be970017cd11fe5b708bde2d3"} Sep 10 02:46:58.304716 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.304689 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4m4vt" event={"ID":"edeb603a-a5ab-4613-9b62-0a59baee7e49","Type":"ContainerStarted","Data":"fa73d932511f1cbc45b0099914e21215112f36ad23f0477aeb148787b09891fd"} Sep 10 02:46:58.306591 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.306569 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="99b9b7183390e1c4629d1fe521056e63e7c3fb395903e70a1880ea06f1b34603" exitCode=0 Sep 10 02:46:58.306679 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.306633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"99b9b7183390e1c4629d1fe521056e63e7c3fb395903e70a1880ea06f1b34603"} Sep 10 02:46:58.310234 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.310204 2583 generic.go:356] "Generic (PLEG): container finished" podID="a619f86e883323954dd3877e7a62c9fd" containerID="d3f60b0bb759c7e6deb1eaa9c31fd7c1942df7d1b8c6c0fed56ed7f84879150f" exitCode=0 Sep 10 02:46:58.310325 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.310311 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" event={"ID":"a619f86e883323954dd3877e7a62c9fd","Type":"ContainerDied","Data":"d3f60b0bb759c7e6deb1eaa9c31fd7c1942df7d1b8c6c0fed56ed7f84879150f"} Sep 10 02:46:58.314934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.314890 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" event={"ID":"a27cb825-d5ee-496d-bf3a-5b0d4cad582f","Type":"ContainerStarted","Data":"b48e668ad9c245bb3408e9fd27e59f299090b5e6bb8582439765ea9c63666da0"} Sep 10 02:46:58.316013 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.315959 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vp6n2" podStartSLOduration=2.755879684 podStartE2EDuration="21.315940227s" totalImagesPullingTime="18.560060543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:58.314661166 +0000 UTC m=+21.838517856" watchObservedRunningTime="2026-09-10 02:46:58.315940227 +0000 UTC m=+21.839796914" Sep 10 02:46:58.319340 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"34679fd4f7374b37f9a13b29a754af71931d53b5994f74e8e77c2b037c7442a9"} Sep 10 02:46:58.319441 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319389 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"8ad63d11ce619d01556422cc731fb10b2cf40d76768e9b5ed93516e541bde862"} Sep 10 02:46:58.319565 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319442 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"8f23954c71e183c413fbf037ee74f3bbd647cd800da479fb3d81d6bae819c2af"} Sep 10 02:46:58.319565 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"23c906558c05fb6e2948a7c52e429d2a265bf4aa113ec10e18b6be708028586f"} Sep 10 02:46:58.319565 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"6b21dea92032d152dab283134b2ef4f2be6051b84e0525f59a2c5411432291e3"} Sep 10 02:46:58.319717 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.319591 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"2e37fa3d9e3e8dcd40ca627ab4d86c3c6481a5fda0eaa77b949cbec794015e64"} Sep 10 02:46:58.359935 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.359452 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4m4vt" podStartSLOduration=2.697505779 podStartE2EDuration="21.359430355s" totalImagesPullingTime="18.661924576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:58.358949634 +0000 UTC m=+21.882806322" watchObservedRunningTime="2026-09-10 02:46:58.359430355 +0000 UTC m=+21.883287042" Sep 10 02:46:58.372445 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.372380 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zx4l6" podStartSLOduration=2.667188807 podStartE2EDuration="21.372359742s" totalImagesPullingTime="18.705170935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:58.371890385 +0000 UTC m=+21.895747074" watchObservedRunningTime="2026-09-10 02:46:58.372359742 +0000 UTC m=+21.896216428" Sep 10 02:46:58.994578 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:58.994545 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:46:58.995276 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:58.994688 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:46:59.104279 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.104244 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 02:46:59.323791 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.323689 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" event={"ID":"a619f86e883323954dd3877e7a62c9fd","Type":"ContainerStarted","Data":"2b69f9c953712819118ce8c1d5279c4f3642de7d3113fc2df02c1daaf1ee1eb2"} Sep 10 02:46:59.325451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.325415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" event={"ID":"a27cb825-d5ee-496d-bf3a-5b0d4cad582f","Type":"ContainerStarted","Data":"8c6a141319ae1603d31823a3ab95af28116f9bfa4c5e67138cb10039e19398be"} Sep 10 02:46:59.326762 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.326735 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2krg" event={"ID":"0ba51178-9185-4c38-ae2d-9f903bae3e0c","Type":"ContainerStarted","Data":"2a9fd44a0fa5ab0c30798603fa2a88038285b98f22ed4ef499dea8429d099066"} Sep 10 02:46:59.340551 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.340486 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-225.ec2.internal" podStartSLOduration=22.340463775 podStartE2EDuration="22.340463775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:59.339924972 +0000 UTC m=+22.863781659" watchObservedRunningTime="2026-09-10 02:46:59.340463775 +0000 UTC m=+22.864320464" Sep 10 02:46:59.352591 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.352541 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g2krg" podStartSLOduration=3.668773889 podStartE2EDuration="22.352526238s" totalImagesPullingTime="18.683752349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:59.352192728 +0000 UTC m=+22.876049416" watchObservedRunningTime="2026-09-10 02:46:59.352526238 +0000 UTC m=+22.876382925" Sep 10 02:46:59.981792 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.981682 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T02:46:59.104265514Z","UUID":"0044a702-2fe9-467a-87c3-e500bf030b48","Handler":null,"Name":"","Endpoint":""} Sep 10 02:46:59.983446 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.983419 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 02:46:59.983582 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.983459 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 02:46:59.994197 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.994169 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:46:59.994334 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:46:59.994170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:46:59.994334 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:59.994279 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:46:59.994462 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:46:59.994383 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:00.335605 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:00.335389 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"8536f2c7d88f0ce49e20b9b8fb491bc0b0e15dabe1ce8667249707f408546277"} Sep 10 02:47:00.994468 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:00.994438 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:00.994637 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:00.994547 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:01.340418 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.340327 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" event={"ID":"a27cb825-d5ee-496d-bf3a-5b0d4cad582f","Type":"ContainerStarted","Data":"6508c3ae92129aa671e4136e716b1102e2030a4eb50d8466bb877082ab6e47b1"} Sep 10 02:47:01.357652 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.357593 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-97bdd" podStartSLOduration=3.313476707 podStartE2EDuration="24.35757741s" totalImagesPullingTime="21.044100703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:01.35672486 +0000 UTC m=+24.880581548" watchObservedRunningTime="2026-09-10 02:47:01.35757741 +0000 UTC m=+24.881434097" Sep 10 02:47:01.545119 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.545077 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:47:01.546314 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.546292 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:47:01.994618 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.994577 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:01.994814 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:01.994590 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:01.994814 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:01.994725 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:47:01.994814 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:01.994778 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:02.343005 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:02.342945 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:47:02.343459 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:02.343360 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4m4vt" Sep 10 02:47:02.993814 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:02.993722 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:02.993996 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:02.993854 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:03.994372 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:03.994157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:03.994719 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:03.994157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:03.994719 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:03.994429 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:47:03.994719 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:03.994470 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:04.350827 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.350732 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" event={"ID":"aea63931-b129-4c99-9a59-f809d68496e3","Type":"ContainerStarted","Data":"a71e397fff1dfb156cb3e98a0c576c75fc6b64d5c0054020ba07910c79e24b7f"} Sep 10 02:47:04.351018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.350999 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:04.351087 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.351077 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:04.351136 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.351127 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:04.352911 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.352884 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="0d0666f0a17e367f8b74efe3a4d159184d11c530b0171371a66f821e9c549b15" exitCode=0 Sep 10 02:47:04.353033 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.352960 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"0d0666f0a17e367f8b74efe3a4d159184d11c530b0171371a66f821e9c549b15"} Sep 10 02:47:04.366928 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.366899 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:04.367058 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.367048 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:04.376131 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.376092 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" podStartSLOduration=8.387727382 podStartE2EDuration="27.376079478s" totalImagesPullingTime="18.988352096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.374989474 +0000 UTC m=+27.898846161" watchObservedRunningTime="2026-09-10 02:47:04.376079478 +0000 UTC m=+27.899936166" Sep 10 02:47:04.994315 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:04.994280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:04.994496 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:04.994422 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:05.326369 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.326322 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vbpc"] Sep 10 02:47:05.329479 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.329450 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dkxnf"] Sep 10 02:47:05.329619 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.329564 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:05.329677 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:05.329656 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:05.330307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.330270 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z98rn"] Sep 10 02:47:05.330431 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.330376 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:05.330510 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:05.330485 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:47:05.355793 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:05.355763 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:05.355966 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:05.355894 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:06.361678 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:06.361649 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="0518c1be078efa115f2b2092be3ff72575e0461559348014bb74b59ccd96d1ec" exitCode=0 Sep 10 02:47:06.362107 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:06.361737 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"0518c1be078efa115f2b2092be3ff72575e0461559348014bb74b59ccd96d1ec"} Sep 10 02:47:06.994763 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:06.994724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:06.994961 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:06.994785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:06.994961 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:06.994890 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:47:06.994961 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:06.994943 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:06.995126 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:06.994987 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:06.995126 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:06.995055 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:08.369060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:08.368877 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="a2b4c9ced8db37f7f9baa28c43d00ba398924a4a4aab7b437b59f70098328550" exitCode=0 Sep 10 02:47:08.369060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:08.368946 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"a2b4c9ced8db37f7f9baa28c43d00ba398924a4a4aab7b437b59f70098328550"} Sep 10 02:47:08.993966 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:08.993932 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:08.994184 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:08.993932 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:08.994184 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:08.994069 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:08.994184 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:08.994061 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vbpc" podUID="307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff" Sep 10 02:47:08.994368 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:08.994177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z98rn" podUID="403bbe5c-24b0-4de7-ac04-0c6fe85e81d0" Sep 10 02:47:08.994368 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:08.994251 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkxnf" podUID="52c15b82-ac94-4fae-a975-195f35311fd2" Sep 10 02:47:09.648359 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:09.648319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:09.648732 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.648451 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:47:09.648732 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.648512 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs podName:403bbe5c-24b0-4de7-ac04-0c6fe85e81d0 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:41.648494545 +0000 UTC m=+65.172351472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs") pod "network-metrics-daemon-z98rn" (UID: "403bbe5c-24b0-4de7-ac04-0c6fe85e81d0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:47:09.749275 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:09.749230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:09.749447 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.749399 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:47:09.749524 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.749464 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret podName:307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff nodeName:}" failed. No retries permitted until 2026-09-10 02:47:41.749445054 +0000 UTC m=+65.273301733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret") pod "global-pull-secret-syncer-7vbpc" (UID: "307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:47:09.849956 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:09.849919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:09.850129 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.850076 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:47:09.850129 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.850099 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:47:09.850129 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.850110 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gfzfg for pod openshift-network-diagnostics/network-check-target-dkxnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:47:09.850294 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:09.850177 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg podName:52c15b82-ac94-4fae-a975-195f35311fd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:41.850157506 +0000 UTC m=+65.374014188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfzfg" (UniqueName: "kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg") pod "network-check-target-dkxnf" (UID: "52c15b82-ac94-4fae-a975-195f35311fd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:47:10.297699 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.297666 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-225.ec2.internal" event="NodeReady" Sep 10 02:47:10.298014 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.297821 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 02:47:10.336320 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.336282 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:47:10.352667 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.352494 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kj7w5"] Sep 10 02:47:10.353018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.352991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.355229 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.355074 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gk6mx\"" Sep 10 02:47:10.355229 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.355151 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 02:47:10.355464 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.355439 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 02:47:10.355630 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.355616 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 02:47:10.361241 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.361217 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 02:47:10.365237 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.365212 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts"] Sep 10 02:47:10.365469 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.365451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.368200 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.368175 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 02:47:10.368443 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.368421 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 02:47:10.368622 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.368600 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 02:47:10.368827 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.368806 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 02:47:10.368949 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.368830 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fvcvm\"" Sep 10 02:47:10.375088 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.375065 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 02:47:10.381973 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.381609 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ff84dbc68-p77s6"] Sep 10 02:47:10.381973 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.381777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.384200 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.384172 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 02:47:10.384635 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.384616 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 02:47:10.385491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.385471 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 02:47:10.386131 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.385827 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8wrq8\"" Sep 10 02:47:10.386288 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.386266 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 02:47:10.393951 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.393928 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:47:10.394069 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.393967 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2mdtx"] Sep 10 02:47:10.394129 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.394079 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.397292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397265 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gnqtz\"" Sep 10 02:47:10.397292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397287 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 02:47:10.397491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397308 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 02:47:10.397695 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397601 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 02:47:10.397695 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397613 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 02:47:10.397695 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397612 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 02:47:10.397906 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.397601 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 02:47:10.408458 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.408428 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pw56k"] Sep 10 02:47:10.408590 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.408569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.411670 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.411637 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 02:47:10.411801 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.411715 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5sdzz\"" Sep 10 02:47:10.411801 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.411724 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 02:47:10.411996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.411641 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 02:47:10.423015 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.422991 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts"] Sep 10 02:47:10.423141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.423033 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kj7w5"] Sep 10 02:47:10.423141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.423055 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mdtx"] Sep 10 02:47:10.423141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.423075 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pw56k"] Sep 10 02:47:10.423141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.423106 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ff84dbc68-p77s6"] Sep 10 02:47:10.423141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.423113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.425403 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.425383 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gq58s\"" Sep 10 02:47:10.425517 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.425403 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 02:47:10.425595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.425406 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 02:47:10.434570 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.434541 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4m4vt_edeb603a-a5ab-4613-9b62-0a59baee7e49/konnectivity-agent/0.log" limit=1024 Sep 10 02:47:10.454997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.454964 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.454997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.454997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/848ff24f-562d-4a9b-be26-609c2b74a313-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-snapshots\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419faac4-860c-472c-b9c7-1fb983cc3085-serving-cert\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzlx\" (UniqueName: \"kubernetes.io/projected/419faac4-860c-472c-b9c7-1fb983cc3085-kube-api-access-xnzlx\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455219 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkm9\" (UniqueName: \"kubernetes.io/projected/848ff24f-562d-4a9b-be26-609c2b74a313-kube-api-access-gnkm9\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455471 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj44x\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.455654 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455486 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-tmp\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.455654 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.455504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.520172 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.520132 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4m4vt_edeb603a-a5ab-4613-9b62-0a59baee7e49/konnectivity-agent/0.log" limit=1024 Sep 10 02:47:10.556314 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.556314 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.556314 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-tmp-dir\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkm9\" (UniqueName: \"kubernetes.io/projected/848ff24f-562d-4a9b-be26-609c2b74a313-kube-api-access-gnkm9\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hcw\" (UniqueName: \"kubernetes.io/projected/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-kube-api-access-g5hcw\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.556533 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cdf66b1-4727-4e1c-977b-448c12a39d00-cert\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj44x\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.556609 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.556606 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:11.056583854 +0000 UTC m=+34.580440522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-tmp\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmm7\" (UniqueName: \"kubernetes.io/projected/1cdf66b1-4727-4e1c-977b-448c12a39d00-kube-api-access-zmmm7\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-stats-auth\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556765 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.556901 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.556915 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.556965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.556974 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:11.056959028 +0000 UTC m=+34.580815719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-config-volume\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.557077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-default-certificate\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwzn\" (UniqueName: \"kubernetes.io/projected/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-kube-api-access-6mwzn\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/848ff24f-562d-4a9b-be26-609c2b74a313-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-snapshots\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419faac4-860c-472c-b9c7-1fb983cc3085-serving-cert\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557306 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzlx\" (UniqueName: \"kubernetes.io/projected/419faac4-860c-472c-b9c7-1fb983cc3085-kube-api-access-xnzlx\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557337 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.557690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-tmp\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.558146 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557713 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.558146 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557749 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.558146 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557929 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/419faac4-860c-472c-b9c7-1fb983cc3085-snapshots\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.558146 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.557955 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.558345 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.558277 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/419faac4-860c-472c-b9c7-1fb983cc3085-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.558345 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.558316 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.558438 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.558381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/848ff24f-562d-4a9b-be26-609c2b74a313-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.562268 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.562148 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/419faac4-860c-472c-b9c7-1fb983cc3085-serving-cert\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.562268 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.562181 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.562268 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.562207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.564880 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.564834 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.565099 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.565073 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj44x\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:10.565362 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.565336 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzlx\" (UniqueName: \"kubernetes.io/projected/419faac4-860c-472c-b9c7-1fb983cc3085-kube-api-access-xnzlx\") pod \"insights-operator-76bcd6ccf5-kj7w5\" (UID: \"419faac4-860c-472c-b9c7-1fb983cc3085\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.565543 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.565520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkm9\" (UniqueName: \"kubernetes.io/projected/848ff24f-562d-4a9b-be26-609c2b74a313-kube-api-access-gnkm9\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:10.658265 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hcw\" (UniqueName: \"kubernetes.io/projected/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-kube-api-access-g5hcw\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.658265 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cdf66b1-4727-4e1c-977b-448c12a39d00-cert\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmm7\" (UniqueName: \"kubernetes.io/projected/1cdf66b1-4727-4e1c-977b-448c12a39d00-kube-api-access-zmmm7\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-stats-auth\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658440 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.658520 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.658564 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:11.158543798 +0000 UTC m=+34.682400466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.658600 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:11.158573251 +0000 UTC m=+34.682429929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-config-volume\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658684 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-default-certificate\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwzn\" (UniqueName: \"kubernetes.io/projected/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-kube-api-access-6mwzn\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.658898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.658813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-tmp-dir\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.659523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.659151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-tmp-dir\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.659523 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.659239 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:10.659523 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:10.659281 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:11.159267792 +0000 UTC m=+34.683124472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:10.659523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.659326 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-config-volume\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.661457 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.661431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-default-certificate\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.661612 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.661431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-stats-auth\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.661612 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.661534 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cdf66b1-4727-4e1c-977b-448c12a39d00-cert\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.670739 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.670703 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hcw\" (UniqueName: \"kubernetes.io/projected/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-kube-api-access-g5hcw\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:10.671108 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.671085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwzn\" (UniqueName: \"kubernetes.io/projected/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-kube-api-access-6mwzn\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:10.671330 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.671312 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmm7\" (UniqueName: \"kubernetes.io/projected/1cdf66b1-4727-4e1c-977b-448c12a39d00-kube-api-access-zmmm7\") pod \"ingress-canary-2mdtx\" (UID: \"1cdf66b1-4727-4e1c-977b-448c12a39d00\") " pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.677588 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.677562 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" Sep 10 02:47:10.722965 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.722927 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mdtx" Sep 10 02:47:10.994155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.994119 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:10.994361 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.994118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:10.994361 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.994119 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:10.997386 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997355 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 02:47:10.997523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997391 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 02:47:10.997523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997399 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 02:47:10.997523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997441 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 02:47:10.997523 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\"" Sep 10 02:47:10.998033 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:10.997837 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\"" Sep 10 02:47:11.000259 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.000238 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kj7w5"] Sep 10 02:47:11.002800 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:11.002774 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419faac4_860c_472c_b9c7_1fb983cc3085.slice/crio-ba2f473d2159c93f51fe125befcfdbf8349d557f17578c3c451926fa421ba4a5 WatchSource:0}: Error finding container ba2f473d2159c93f51fe125befcfdbf8349d557f17578c3c451926fa421ba4a5: Status 404 returned error can't find the container with id ba2f473d2159c93f51fe125befcfdbf8349d557f17578c3c451926fa421ba4a5 Sep 10 02:47:11.021901 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.021826 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mdtx"] Sep 10 02:47:11.026285 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:11.026251 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdf66b1_4727_4e1c_977b_448c12a39d00.slice/crio-0faaf52a5b7a6cc2c81e454367fc0fdb9250d8a04e66f95e2fb49b3da797a550 WatchSource:0}: Error finding container 0faaf52a5b7a6cc2c81e454367fc0fdb9250d8a04e66f95e2fb49b3da797a550: Status 404 returned error can't find the container with id 0faaf52a5b7a6cc2c81e454367fc0fdb9250d8a04e66f95e2fb49b3da797a550 Sep 10 02:47:11.062442 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.062411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:11.062648 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.062556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:11.062648 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.062580 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:11.062648 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.062603 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:11.062809 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.062681 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:11.062809 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.062697 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:12.062648853 +0000 UTC m=+35.586505525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:11.062809 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.062743 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:12.062727858 +0000 UTC m=+35.586584523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:11.163221 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.163137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:11.163221 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.163175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.163287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.163327 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.163392 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.163407 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:12.163385768 +0000 UTC m=+35.687242446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.163437 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:12.163420505 +0000 UTC m=+35.687277174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:11.163478 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:11.163457 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:12.163446955 +0000 UTC m=+35.687303621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:11.377792 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.377754 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" event={"ID":"419faac4-860c-472c-b9c7-1fb983cc3085","Type":"ContainerStarted","Data":"ba2f473d2159c93f51fe125befcfdbf8349d557f17578c3c451926fa421ba4a5"} Sep 10 02:47:11.378841 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:11.378816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mdtx" event={"ID":"1cdf66b1-4727-4e1c-977b-448c12a39d00","Type":"ContainerStarted","Data":"0faaf52a5b7a6cc2c81e454367fc0fdb9250d8a04e66f95e2fb49b3da797a550"} Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:12.070739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:12.070820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.070901 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.070969 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.07094875 +0000 UTC m=+37.594805420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.070990 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.071002 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:12.071060 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.071034 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.071023906 +0000 UTC m=+37.594880585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:12.172212 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:12.172301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:12.172340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.172379 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.172442 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.17242564 +0000 UTC m=+37.696282310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.172459 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.172449552 +0000 UTC m=+37.696306231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.172515 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:12.172569 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:12.172546 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.172535852 +0000 UTC m=+37.696392532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:13.917224 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.917056 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs"] Sep 10 02:47:13.930100 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.930070 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt"] Sep 10 02:47:13.930258 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.930224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938082 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938107 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938408 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938470 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938602 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-s7wz6\"" Sep 10 02:47:13.938780 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938637 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:13.939696 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.938810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs"] Sep 10 02:47:13.943935 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.943909 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:47:13.944919 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.944894 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9sv9j\"" Sep 10 02:47:13.945109 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.945086 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 02:47:13.945367 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.945348 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 02:47:13.950227 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.950204 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt"] Sep 10 02:47:13.962324 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.962291 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp"] Sep 10 02:47:13.976192 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.976161 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp"] Sep 10 02:47:13.976377 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.976276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" Sep 10 02:47:13.979369 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.979150 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:47:13.979369 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.979285 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-75s5q\"" Sep 10 02:47:13.979369 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.979307 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 02:47:13.986412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.986389 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k"] Sep 10 02:47:13.992934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.992909 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:13.995079 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.995057 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 02:47:13.995318 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.995297 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vtrdr"] Sep 10 02:47:13.995463 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.995442 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 02:47:13.995463 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.995453 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v76rn\"" Sep 10 02:47:13.995623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.995554 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 02:47:13.998877 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:13.998836 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:47:14.004613 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.003810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k"] Sep 10 02:47:14.004613 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.003984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.007284 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.007046 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 02:47:14.007284 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.007096 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:47:14.007284 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.007103 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 02:47:14.007284 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.007207 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dtj4c\"" Sep 10 02:47:14.007617 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.007379 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 02:47:14.011692 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.011668 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vtrdr"] Sep 10 02:47:14.012697 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.012672 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 02:47:14.088748 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088718 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1c1665-a796-4a58-8798-14a539c3e8fd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.088913 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088760 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec1c1665-a796-4a58-8798-14a539c3e8fd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.088913 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jj4n\" (UniqueName: \"kubernetes.io/projected/a5acc330-46a8-4ae6-91e3-1e1a89503ecd-kube-api-access-9jj4n\") pod \"volume-data-source-validator-695c895bd6-tp8vp\" (UID: \"a5acc330-46a8-4ae6-91e3-1e1a89503ecd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" Sep 10 02:47:14.088913 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-config\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.089018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088964 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26fd4e22-d22f-429e-b429-a4c64cdc789d-config\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.089018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.088992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be0e6efc-131e-4866-851c-27b681b19cfd-serving-cert\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.089078 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh25c\" (UniqueName: \"kubernetes.io/projected/ec1c1665-a796-4a58-8798-14a539c3e8fd-kube-api-access-dh25c\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.089078 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-trusted-ca\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.089141 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:14.089172 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26fd4e22-d22f-429e-b429-a4c64cdc789d-serving-cert\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.089172 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqcr\" (UniqueName: \"kubernetes.io/projected/be0e6efc-131e-4866-851c-27b681b19cfd-kube-api-access-bqqcr\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.089232 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.089172 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:14.089232 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.089182 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:14.089232 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbstp\" (UniqueName: \"kubernetes.io/projected/26fd4e22-d22f-429e-b429-a4c64cdc789d-kube-api-access-kbstp\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.089232 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9rx\" (UniqueName: \"kubernetes.io/projected/206660d5-90d2-47b7-9392-c82a6973f90e-kube-api-access-zs9rx\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.089232 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.089232 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.089211335 +0000 UTC m=+41.613068018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:14.089385 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.089385 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.089368 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:14.089486 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.089473 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:14.089935 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.089924 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.089507631 +0000 UTC m=+41.613364296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:14.190169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1c1665-a796-4a58-8798-14a539c3e8fd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.190169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec1c1665-a796-4a58-8798-14a539c3e8fd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.190169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jj4n\" (UniqueName: \"kubernetes.io/projected/a5acc330-46a8-4ae6-91e3-1e1a89503ecd-kube-api-access-9jj4n\") pod \"volume-data-source-validator-695c895bd6-tp8vp\" (UID: \"a5acc330-46a8-4ae6-91e3-1e1a89503ecd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-config\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26fd4e22-d22f-429e-b429-a4c64cdc789d-config\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be0e6efc-131e-4866-851c-27b681b19cfd-serving-cert\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh25c\" (UniqueName: \"kubernetes.io/projected/ec1c1665-a796-4a58-8798-14a539c3e8fd-kube-api-access-dh25c\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190379 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-trusted-ca\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.190452 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190426 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26fd4e22-d22f-429e-b429-a4c64cdc789d-serving-cert\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqcr\" (UniqueName: \"kubernetes.io/projected/be0e6efc-131e-4866-851c-27b681b19cfd-kube-api-access-bqqcr\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbstp\" (UniqueName: \"kubernetes.io/projected/26fd4e22-d22f-429e-b429-a4c64cdc789d-kube-api-access-kbstp\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190561 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9rx\" (UniqueName: \"kubernetes.io/projected/206660d5-90d2-47b7-9392-c82a6973f90e-kube-api-access-zs9rx\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190602 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.190785 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.190790 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190805 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec1c1665-a796-4a58-8798-14a539c3e8fd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.190841 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.190824651 +0000 UTC m=+41.714681335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190907 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26fd4e22-d22f-429e-b429-a4c64cdc789d-config\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.190926 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.190943 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-config\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.190963 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.190950619 +0000 UTC m=+41.714807285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:14.191142 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.191113 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.19109944 +0000 UTC m=+41.714956105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:14.191482 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.191215 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:47:14.191482 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.191281 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls podName:206660d5-90d2-47b7-9392-c82a6973f90e nodeName:}" failed. No retries permitted until 2026-09-10 02:47:14.691258803 +0000 UTC m=+38.215115484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wf9jt" (UID: "206660d5-90d2-47b7-9392-c82a6973f90e") : secret "samples-operator-tls" not found Sep 10 02:47:14.191482 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.191322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0e6efc-131e-4866-851c-27b681b19cfd-trusted-ca\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.193596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.193569 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26fd4e22-d22f-429e-b429-a4c64cdc789d-serving-cert\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.193724 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.193619 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1c1665-a796-4a58-8798-14a539c3e8fd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.193724 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.193676 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be0e6efc-131e-4866-851c-27b681b19cfd-serving-cert\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.200626 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.200592 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh25c\" (UniqueName: \"kubernetes.io/projected/ec1c1665-a796-4a58-8798-14a539c3e8fd-kube-api-access-dh25c\") pod \"kube-storage-version-migrator-operator-8d7dd4675-sdjcs\" (UID: \"ec1c1665-a796-4a58-8798-14a539c3e8fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.200757 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.200688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jj4n\" (UniqueName: \"kubernetes.io/projected/a5acc330-46a8-4ae6-91e3-1e1a89503ecd-kube-api-access-9jj4n\") pod \"volume-data-source-validator-695c895bd6-tp8vp\" (UID: \"a5acc330-46a8-4ae6-91e3-1e1a89503ecd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" Sep 10 02:47:14.200835 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.200815 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbstp\" (UniqueName: \"kubernetes.io/projected/26fd4e22-d22f-429e-b429-a4c64cdc789d-kube-api-access-kbstp\") pod \"service-ca-operator-859f94c4b-2wn9k\" (UID: \"26fd4e22-d22f-429e-b429-a4c64cdc789d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.201316 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.201295 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9rx\" (UniqueName: \"kubernetes.io/projected/206660d5-90d2-47b7-9392-c82a6973f90e-kube-api-access-zs9rx\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.201522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.201506 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqcr\" (UniqueName: \"kubernetes.io/projected/be0e6efc-131e-4866-851c-27b681b19cfd-kube-api-access-bqqcr\") pod \"console-operator-85b454874b-vtrdr\" (UID: \"be0e6efc-131e-4866-851c-27b681b19cfd\") " pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.242225 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.242187 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" Sep 10 02:47:14.288175 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.288143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" Sep 10 02:47:14.307640 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.307604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" Sep 10 02:47:14.318100 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.318062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:14.694464 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:14.694425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:14.694660 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.694596 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:47:14.694722 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:14.694676 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls podName:206660d5-90d2-47b7-9392-c82a6973f90e nodeName:}" failed. No retries permitted until 2026-09-10 02:47:15.694654045 +0000 UTC m=+39.218510712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wf9jt" (UID: "206660d5-90d2-47b7-9392-c82a6973f90e") : secret "samples-operator-tls" not found Sep 10 02:47:15.685997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:15.685744 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs"] Sep 10 02:47:15.690005 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:15.689977 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp"] Sep 10 02:47:15.694295 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:15.694189 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vtrdr"] Sep 10 02:47:15.704193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:15.704084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:15.704358 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:15.704243 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:47:15.704358 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:15.704321 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls podName:206660d5-90d2-47b7-9392-c82a6973f90e nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.704300295 +0000 UTC m=+41.228156974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wf9jt" (UID: "206660d5-90d2-47b7-9392-c82a6973f90e") : secret "samples-operator-tls" not found Sep 10 02:47:15.712350 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:15.712261 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k"] Sep 10 02:47:15.716290 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:15.716249 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0e6efc_131e_4866_851c_27b681b19cfd.slice/crio-cd0554c6d283e2bf173abba1d7cbfd868b5c3ef8863ebeacfa582acc037a2944 WatchSource:0}: Error finding container cd0554c6d283e2bf173abba1d7cbfd868b5c3ef8863ebeacfa582acc037a2944: Status 404 returned error can't find the container with id cd0554c6d283e2bf173abba1d7cbfd868b5c3ef8863ebeacfa582acc037a2944 Sep 10 02:47:15.717338 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:15.717312 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5acc330_46a8_4ae6_91e3_1e1a89503ecd.slice/crio-3f54e7ed7203bcc8d0585b0601400f2169909032927c3aa0d14c0a5eeaf553ac WatchSource:0}: Error finding container 3f54e7ed7203bcc8d0585b0601400f2169909032927c3aa0d14c0a5eeaf553ac: Status 404 returned error can't find the container with id 3f54e7ed7203bcc8d0585b0601400f2169909032927c3aa0d14c0a5eeaf553ac Sep 10 02:47:15.720781 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:15.720120 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1c1665_a796_4a58_8798_14a539c3e8fd.slice/crio-6458d2e2efa1163f56b9f9953f9e8e534a21d28f130a7f8648b1235ee8b17fa2 WatchSource:0}: Error finding container 6458d2e2efa1163f56b9f9953f9e8e534a21d28f130a7f8648b1235ee8b17fa2: Status 404 returned error can't find the container with id 6458d2e2efa1163f56b9f9953f9e8e534a21d28f130a7f8648b1235ee8b17fa2 Sep 10 02:47:16.398193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.398025 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="0ccbe28354332a63757a17d750325a801a2e796caec2a818182231bea4e17435" exitCode=0 Sep 10 02:47:16.398193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.398129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"0ccbe28354332a63757a17d750325a801a2e796caec2a818182231bea4e17435"} Sep 10 02:47:16.402936 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.402876 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mdtx" event={"ID":"1cdf66b1-4727-4e1c-977b-448c12a39d00","Type":"ContainerStarted","Data":"7be732389b9fbf8ae5f508ae6ec03627f00d9aaea147748a34094b45aacc89ac"} Sep 10 02:47:16.408120 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.407316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" event={"ID":"419faac4-860c-472c-b9c7-1fb983cc3085","Type":"ContainerStarted","Data":"4a0dfd3d178ea0d454fabad801ca6dacde0c8cd10dc25b8da3140f5dd7788555"} Sep 10 02:47:16.410222 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.410163 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" event={"ID":"ec1c1665-a796-4a58-8798-14a539c3e8fd","Type":"ContainerStarted","Data":"6458d2e2efa1163f56b9f9953f9e8e534a21d28f130a7f8648b1235ee8b17fa2"} Sep 10 02:47:16.413072 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.413036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" event={"ID":"a5acc330-46a8-4ae6-91e3-1e1a89503ecd","Type":"ContainerStarted","Data":"3f54e7ed7203bcc8d0585b0601400f2169909032927c3aa0d14c0a5eeaf553ac"} Sep 10 02:47:16.414968 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.414908 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" event={"ID":"be0e6efc-131e-4866-851c-27b681b19cfd","Type":"ContainerStarted","Data":"cd0554c6d283e2bf173abba1d7cbfd868b5c3ef8863ebeacfa582acc037a2944"} Sep 10 02:47:16.416481 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.416457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" event={"ID":"26fd4e22-d22f-429e-b429-a4c64cdc789d","Type":"ContainerStarted","Data":"3fcb01756d9cc456c51f3b62d0df7a337664e65da25bb510667d4e40027cc72b"} Sep 10 02:47:16.448701 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.448566 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2mdtx" podStartSLOduration=2.066574557 podStartE2EDuration="6.448543036s" totalImagesPullingTime="4.381968479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:16.433024202 +0000 UTC m=+39.956880891" watchObservedRunningTime="2026-09-10 02:47:16.448543036 +0000 UTC m=+39.972399729" Sep 10 02:47:16.658077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.658011 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" podStartSLOduration=9.256330836 podStartE2EDuration="13.657982122s" totalImagesPullingTime="4.401651286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:16.448804673 +0000 UTC m=+39.972661360" watchObservedRunningTime="2026-09-10 02:47:16.657982122 +0000 UTC m=+40.181838809" Sep 10 02:47:16.658614 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.658584 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-77jcb"] Sep 10 02:47:16.676557 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.675647 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77jcb"] Sep 10 02:47:16.676557 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.675787 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.681158 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.681074 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fq77l\"" Sep 10 02:47:16.682024 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.681419 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 02:47:16.682991 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.682972 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 02:47:16.813332 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.812919 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a116314-9fb9-4786-b01a-1bbb80aef26a-data-volume\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.813332 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.813063 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.813332 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.813133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7gl\" (UniqueName: \"kubernetes.io/projected/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-api-access-xk7gl\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.813332 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.813164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.813332 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.813230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a116314-9fb9-4786-b01a-1bbb80aef26a-crio-socket\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.914624 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.914527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7gl\" (UniqueName: \"kubernetes.io/projected/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-api-access-xk7gl\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.914624 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.914586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.914831 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.914658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a116314-9fb9-4786-b01a-1bbb80aef26a-crio-socket\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.914831 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.914691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a116314-9fb9-4786-b01a-1bbb80aef26a-data-volume\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.914831 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.914796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.915019 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:16.914956 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:16.915019 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:16.915019 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.415000515 +0000 UTC m=+40.938857187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:16.915399 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:16.915236 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:47:16.915399 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:16.915320 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.415295128 +0000 UTC m=+40.939151800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : configmap "kube-rbac-proxy" not found Sep 10 02:47:16.915399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.915354 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a116314-9fb9-4786-b01a-1bbb80aef26a-crio-socket\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.915591 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.915568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a116314-9fb9-4786-b01a-1bbb80aef26a-data-volume\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:16.924692 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:16.924642 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7gl\" (UniqueName: \"kubernetes.io/projected/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-api-access-xk7gl\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:17.420048 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:17.420010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:17.420283 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.420160 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:47:17.420283 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:17.420178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:17.420283 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.420250 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.420227842 +0000 UTC m=+41.944084530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : configmap "kube-rbac-proxy" not found Sep 10 02:47:17.420535 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.420301 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:17.420535 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.420367 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.420353512 +0000 UTC m=+41.944210177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:17.425163 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:17.425119 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a0063e7-0b15-4581-b2ca-776e53c51b21" containerID="1cab10233fed8106fae40a14fd0142adb944b56cb8bfa5fd3d4154205a5fd816" exitCode=0 Sep 10 02:47:17.425318 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:17.425256 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerDied","Data":"1cab10233fed8106fae40a14fd0142adb944b56cb8bfa5fd3d4154205a5fd816"} Sep 10 02:47:17.723784 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:17.723706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:17.723979 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.723876 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:47:17.723979 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:17.723948 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls podName:206660d5-90d2-47b7-9392-c82a6973f90e nodeName:}" failed. No retries permitted until 2026-09-10 02:47:21.723930745 +0000 UTC m=+45.247787429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wf9jt" (UID: "206660d5-90d2-47b7-9392-c82a6973f90e") : secret "samples-operator-tls" not found Sep 10 02:47:18.128185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.128132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.128233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.128298 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.128320 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.128394 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:26.128370612 +0000 UTC m=+49.652227289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.128415 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:18.128649 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.128490 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:26.128470559 +0000 UTC m=+49.652327227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:18.229131 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.229066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:18.229278 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.229146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:18.229278 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.229217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:18.229369 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.229281 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:18.229369 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.229304 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:26.229274109 +0000 UTC m=+49.753130787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:18.229369 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.229342 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:26.229326465 +0000 UTC m=+49.753183150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:18.229369 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.229346 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:18.229522 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.229399 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:26.229384029 +0000 UTC m=+49.753240703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:18.430966 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.430888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:18.431112 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.430988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:18.431112 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.431036 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:18.431112 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.431073 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:47:18.431112 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.431113 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:20.43109139 +0000 UTC m=+43.954948062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:18.431314 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:18.431131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:20.431121907 +0000 UTC m=+43.954978571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : configmap "kube-rbac-proxy" not found Sep 10 02:47:18.431998 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.431973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb84l" event={"ID":"2a0063e7-0b15-4581-b2ca-776e53c51b21","Type":"ContainerStarted","Data":"245e03f93d3b2f09e3ca29c19da580f95ec53a2e8d7da6266b55c43da3fe6e6a"} Sep 10 02:47:18.452116 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.452054 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bb84l" podStartSLOduration=12.376337119 podStartE2EDuration="41.452034032s" totalImagesPullingTime="29.075696913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:18.451502854 +0000 UTC m=+41.975359545" watchObservedRunningTime="2026-09-10 02:47:18.452034032 +0000 UTC m=+41.975890721" Sep 10 02:47:18.539282 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:18.539257 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp6n2_9f135ece-4051-422d-a794-e4ac35d8c8b7/dns-node-resolver/0.log" Sep 10 02:47:19.337621 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:19.337594 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zx4l6_0ab99a0f-f0bc-4500-b53e-5b10a6cfba07/node-ca/0.log" Sep 10 02:47:19.737992 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:19.737924 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mdtx_1cdf66b1-4727-4e1c-977b-448c12a39d00/serve-healthcheck-canary/0.log" Sep 10 02:47:20.441480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.441444 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/0.log" Sep 10 02:47:20.441962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.441494 2583 generic.go:356] "Generic (PLEG): container finished" podID="be0e6efc-131e-4866-851c-27b681b19cfd" containerID="48e9930db1972c517b40f6c21fc2eb097150e1a25503bc7e6295d7f990c230fc" exitCode=255 Sep 10 02:47:20.441962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.441573 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" event={"ID":"be0e6efc-131e-4866-851c-27b681b19cfd","Type":"ContainerDied","Data":"48e9930db1972c517b40f6c21fc2eb097150e1a25503bc7e6295d7f990c230fc"} Sep 10 02:47:20.441962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.441944 2583 scope.go:117] "RemoveContainer" containerID="48e9930db1972c517b40f6c21fc2eb097150e1a25503bc7e6295d7f990c230fc" Sep 10 02:47:20.443383 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.443354 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" event={"ID":"26fd4e22-d22f-429e-b429-a4c64cdc789d","Type":"ContainerStarted","Data":"d174f50b5843ea39b578007772e4767a12f74eed03c05cf9098744a6b78fc83d"} Sep 10 02:47:20.445137 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.445113 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" event={"ID":"ec1c1665-a796-4a58-8798-14a539c3e8fd","Type":"ContainerStarted","Data":"74e1726db8c130a8c41e3362687389e7caa7d1ffcbae000373b2a3bfca80547f"} Sep 10 02:47:20.446681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.446657 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" event={"ID":"a5acc330-46a8-4ae6-91e3-1e1a89503ecd","Type":"ContainerStarted","Data":"0d1e7fd5c4f21d82523093c61a1f2b592ee7642810365f4179fdba1abd51b92e"} Sep 10 02:47:20.448099 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.448078 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:20.448161 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.448148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:20.448228 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:20.448214 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:20.448284 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:20.448263 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.448250892 +0000 UTC m=+47.972107557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:20.448350 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:20.448281 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:47:20.448440 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:20.448421 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.448367605 +0000 UTC m=+47.972224289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : configmap "kube-rbac-proxy" not found Sep 10 02:47:20.472267 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.472210 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" podStartSLOduration=3.192204128 podStartE2EDuration="7.472186109s" totalImagesPullingTime="4.279981981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:20.47054919 +0000 UTC m=+43.994405893" watchObservedRunningTime="2026-09-10 02:47:20.472186109 +0000 UTC m=+43.996042797" Sep 10 02:47:20.484427 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.484346 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" podStartSLOduration=3.210274204 podStartE2EDuration="7.484319058s" totalImagesPullingTime="4.274044854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:20.483755161 +0000 UTC m=+44.007611848" watchObservedRunningTime="2026-09-10 02:47:20.484319058 +0000 UTC m=+44.008175746" Sep 10 02:47:20.497654 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:20.497523 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tp8vp" podStartSLOduration=3.223984565 podStartE2EDuration="7.497504294s" totalImagesPullingTime="4.273519729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:20.4971612 +0000 UTC m=+44.021017887" watchObservedRunningTime="2026-09-10 02:47:20.497504294 +0000 UTC m=+44.021360985" Sep 10 02:47:21.452187 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452157 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/1.log" Sep 10 02:47:21.452649 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452592 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/0.log" Sep 10 02:47:21.452724 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452627 2583 generic.go:356] "Generic (PLEG): container finished" podID="be0e6efc-131e-4866-851c-27b681b19cfd" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" exitCode=255 Sep 10 02:47:21.452724 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" event={"ID":"be0e6efc-131e-4866-851c-27b681b19cfd","Type":"ContainerDied","Data":"5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443"} Sep 10 02:47:21.452814 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452735 2583 scope.go:117] "RemoveContainer" containerID="48e9930db1972c517b40f6c21fc2eb097150e1a25503bc7e6295d7f990c230fc" Sep 10 02:47:21.452983 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.452962 2583 scope.go:117] "RemoveContainer" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" Sep 10 02:47:21.453186 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:21.453166 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:21.759048 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:21.759019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:21.759197 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:21.759139 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:47:21.759197 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:21.759194 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls podName:206660d5-90d2-47b7-9392-c82a6973f90e nodeName:}" failed. No retries permitted until 2026-09-10 02:47:29.759181172 +0000 UTC m=+53.283037836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-wf9jt" (UID: "206660d5-90d2-47b7-9392-c82a6973f90e") : secret "samples-operator-tls" not found Sep 10 02:47:22.461945 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:22.461915 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/1.log" Sep 10 02:47:22.462427 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:22.462330 2583 scope.go:117] "RemoveContainer" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" Sep 10 02:47:22.462541 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:22.462518 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:24.318587 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:24.318543 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:24.319157 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:24.318609 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:24.319157 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:24.318954 2583 scope.go:117] "RemoveContainer" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" Sep 10 02:47:24.319157 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:24.319096 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:24.484058 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:24.484017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:24.484243 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:24.484130 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:47:24.484243 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:24.484198 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:32.484180336 +0000 UTC m=+56.008037022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : configmap "kube-rbac-proxy" not found Sep 10 02:47:24.484346 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:24.484300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:24.484450 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:24.484433 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:24.484498 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:24.484490 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:32.48447696 +0000 UTC m=+56.008333630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:26.200073 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:26.200035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:26.200109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.200186 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.200207 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5896688487-gvlnt: secret "image-registry-tls" not found Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.200251 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.200260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls podName:e8a2bf3b-5c37-4947-b77d-bc2747b2c224 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:42.200244773 +0000 UTC m=+65.724101442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls") pod "image-registry-5896688487-gvlnt" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224") : secret "image-registry-tls" not found Sep 10 02:47:26.200480 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.200308 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls podName:848ff24f-562d-4a9b-be26-609c2b74a313 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:42.200296159 +0000 UTC m=+65.724152825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-vblts" (UID: "848ff24f-562d-4a9b-be26-609c2b74a313") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:47:26.301185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:26.301147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:26.301244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:26.301272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.301330 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.301373 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:42.301360949 +0000 UTC m=+65.825217614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : configmap references non-existent config key: service-ca.crt Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.301373 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:47:26.301390 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.301387 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls podName:0388e7f9-1f22-4244-9dfb-e6cf87020d8c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:42.301381145 +0000 UTC m=+65.825237810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls") pod "dns-default-pw56k" (UID: "0388e7f9-1f22-4244-9dfb-e6cf87020d8c") : secret "dns-default-metrics-tls" not found Sep 10 02:47:26.301689 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:26.301423 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs podName:3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:42.301407612 +0000 UTC m=+65.825264282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs") pod "router-default-ff84dbc68-p77s6" (UID: "3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46") : secret "router-metrics-certs-default" not found Sep 10 02:47:29.836541 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:29.836481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:29.838887 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:29.838848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/206660d5-90d2-47b7-9392-c82a6973f90e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-wf9jt\" (UID: \"206660d5-90d2-47b7-9392-c82a6973f90e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:29.850690 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:29.850664 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" Sep 10 02:47:30.119611 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:30.119541 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt"] Sep 10 02:47:30.486816 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:30.486725 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" event={"ID":"206660d5-90d2-47b7-9392-c82a6973f90e","Type":"ContainerStarted","Data":"92eef259666d6c70a1734750ea6490754a4d56789d4bf430d527e2f7b75e565f"} Sep 10 02:47:32.496363 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.496320 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" event={"ID":"206660d5-90d2-47b7-9392-c82a6973f90e","Type":"ContainerStarted","Data":"36310fc38593fbaf8a1a15f36656c0f5a48519a4d391d052e6fac9495d4f86d5"} Sep 10 02:47:32.496761 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.496390 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" event={"ID":"206660d5-90d2-47b7-9392-c82a6973f90e","Type":"ContainerStarted","Data":"1b9a127e24626a7e45ad41957a5bed6b0f603ce7e0f5afb120ede42141a7227b"} Sep 10 02:47:32.511896 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.511830 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-wf9jt" podStartSLOduration=17.721702434 podStartE2EDuration="19.51181279s" totalImagesPullingTime="1.790110356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:32.510764543 +0000 UTC m=+56.034621231" watchObservedRunningTime="2026-09-10 02:47:32.51181279 +0000 UTC m=+56.035669477" Sep 10 02:47:32.563115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.563072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:32.563312 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.563243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:32.563364 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:32.563342 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:47:32.563425 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:32.563412 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls podName:1a116314-9fb9-4786-b01a-1bbb80aef26a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:48.563393945 +0000 UTC m=+72.087250629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77jcb" (UID: "1a116314-9fb9-4786-b01a-1bbb80aef26a") : secret "insights-runtime-extractor-tls" not found Sep 10 02:47:32.563738 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:32.563716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a116314-9fb9-4786-b01a-1bbb80aef26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:36.381759 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:36.381729 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d47m4" Sep 10 02:47:39.994067 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:39.994026 2583 scope.go:117] "RemoveContainer" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" Sep 10 02:47:40.525600 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.525574 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:47:40.525959 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.525946 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/1.log" Sep 10 02:47:40.526017 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.525976 2583 generic.go:356] "Generic (PLEG): container finished" podID="be0e6efc-131e-4866-851c-27b681b19cfd" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" exitCode=255 Sep 10 02:47:40.526058 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.526018 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" event={"ID":"be0e6efc-131e-4866-851c-27b681b19cfd","Type":"ContainerDied","Data":"6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4"} Sep 10 02:47:40.526092 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.526060 2583 scope.go:117] "RemoveContainer" containerID="5f28845b21766210c67800c38ef664cfd81c4d64dc27354a89a4a6ec59eb1443" Sep 10 02:47:40.526367 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:40.526343 2583 scope.go:117] "RemoveContainer" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" Sep 10 02:47:40.526528 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:40.526509 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:41.531181 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.531151 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:47:41.743176 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.743125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:41.745343 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.745325 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 02:47:41.756187 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.756166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403bbe5c-24b0-4de7-ac04-0c6fe85e81d0-metrics-certs\") pod \"network-metrics-daemon-z98rn\" (UID: \"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0\") " pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:41.844005 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.843928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:41.846256 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.846235 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 02:47:41.857117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.857095 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff-original-pull-secret\") pod \"global-pull-secret-syncer-7vbpc\" (UID: \"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff\") " pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:41.908847 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.908807 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vbpc" Sep 10 02:47:41.925149 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.925125 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\"" Sep 10 02:47:41.931172 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.931144 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z98rn" Sep 10 02:47:41.944847 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.944801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:41.948738 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.948510 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 02:47:41.960768 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.960728 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 02:47:41.969053 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:41.969025 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzfg\" (UniqueName: \"kubernetes.io/projected/52c15b82-ac94-4fae-a975-195f35311fd2-kube-api-access-gfzfg\") pod \"network-check-target-dkxnf\" (UID: \"52c15b82-ac94-4fae-a975-195f35311fd2\") " pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:42.181503 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.181421 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vbpc"] Sep 10 02:47:42.184745 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.184721 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307f0a5e_ff8f_47b1_9bce_c6f98ac1c4ff.slice/crio-5a4a48622dd91d4ccecaa03a756c564cd0a60bb90ce9cddf8ae884a4ea996849 WatchSource:0}: Error finding container 5a4a48622dd91d4ccecaa03a756c564cd0a60bb90ce9cddf8ae884a4ea996849: Status 404 returned error can't find the container with id 5a4a48622dd91d4ccecaa03a756c564cd0a60bb90ce9cddf8ae884a4ea996849 Sep 10 02:47:42.210453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.210424 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z98rn"] Sep 10 02:47:42.213574 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.213544 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403bbe5c_24b0_4de7_ac04_0c6fe85e81d0.slice/crio-6f22cf79ec35e8979d5c7ca4ccea9e840b0c05aa3f4e1ad9e22086c18aae8667 WatchSource:0}: Error finding container 6f22cf79ec35e8979d5c7ca4ccea9e840b0c05aa3f4e1ad9e22086c18aae8667: Status 404 returned error can't find the container with id 6f22cf79ec35e8979d5c7ca4ccea9e840b0c05aa3f4e1ad9e22086c18aae8667 Sep 10 02:47:42.217557 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.217539 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\"" Sep 10 02:47:42.226506 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.226486 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:42.247881 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.247844 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:42.247977 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.247906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:42.250211 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.250190 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"image-registry-5896688487-gvlnt\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:42.250305 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.250224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/848ff24f-562d-4a9b-be26-609c2b74a313-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-vblts\" (UID: \"848ff24f-562d-4a9b-be26-609c2b74a313\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:42.348623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.348576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:42.348623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.348625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:42.348834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.348654 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:42.349315 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.349289 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-service-ca-bundle\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:42.350950 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.350927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0388e7f9-1f22-4244-9dfb-e6cf87020d8c-metrics-tls\") pod \"dns-default-pw56k\" (UID: \"0388e7f9-1f22-4244-9dfb-e6cf87020d8c\") " pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:42.351064 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.351044 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46-metrics-certs\") pod \"router-default-ff84dbc68-p77s6\" (UID: \"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46\") " pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:42.468112 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.468029 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gk6mx\"" Sep 10 02:47:42.475753 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.475722 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:42.495644 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.495617 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8wrq8\"" Sep 10 02:47:42.498933 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.498905 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dkxnf"] Sep 10 02:47:42.502044 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.502019 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c15b82_ac94_4fae_a975_195f35311fd2.slice/crio-8c89933c1a3746664244b494ec067cade7168feb9d5bf3d0269dada8dc529fa6 WatchSource:0}: Error finding container 8c89933c1a3746664244b494ec067cade7168feb9d5bf3d0269dada8dc529fa6: Status 404 returned error can't find the container with id 8c89933c1a3746664244b494ec067cade7168feb9d5bf3d0269dada8dc529fa6 Sep 10 02:47:42.503646 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.503617 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" Sep 10 02:47:42.506385 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.506368 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gnqtz\"" Sep 10 02:47:42.514486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.514462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:42.538000 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.535532 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gq58s\"" Sep 10 02:47:42.540103 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.539316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z98rn" event={"ID":"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0","Type":"ContainerStarted","Data":"6f22cf79ec35e8979d5c7ca4ccea9e840b0c05aa3f4e1ad9e22086c18aae8667"} Sep 10 02:47:42.543143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.541486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vbpc" event={"ID":"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff","Type":"ContainerStarted","Data":"5a4a48622dd91d4ccecaa03a756c564cd0a60bb90ce9cddf8ae884a4ea996849"} Sep 10 02:47:42.543143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.543029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dkxnf" event={"ID":"52c15b82-ac94-4fae-a975-195f35311fd2","Type":"ContainerStarted","Data":"8c89933c1a3746664244b494ec067cade7168feb9d5bf3d0269dada8dc529fa6"} Sep 10 02:47:42.544300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.544221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:42.757071 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.757042 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:47:42.760959 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.760926 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a2bf3b_5c37_4947_b77d_bc2747b2c224.slice/crio-bcd21c5ae6d1022e7aaa6a1c3f086faf9fbecca1c0ffbb3b7f93eb0ee4039b27 WatchSource:0}: Error finding container bcd21c5ae6d1022e7aaa6a1c3f086faf9fbecca1c0ffbb3b7f93eb0ee4039b27: Status 404 returned error can't find the container with id bcd21c5ae6d1022e7aaa6a1c3f086faf9fbecca1c0ffbb3b7f93eb0ee4039b27 Sep 10 02:47:42.788016 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.787981 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts"] Sep 10 02:47:42.792140 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.792087 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848ff24f_562d_4a9b_be26_609c2b74a313.slice/crio-84cbc78eaac40b30e15be2c766540785f60448bddfb9954592c1e4a4d7e79df5 WatchSource:0}: Error finding container 84cbc78eaac40b30e15be2c766540785f60448bddfb9954592c1e4a4d7e79df5: Status 404 returned error can't find the container with id 84cbc78eaac40b30e15be2c766540785f60448bddfb9954592c1e4a4d7e79df5 Sep 10 02:47:42.810310 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.810090 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ff84dbc68-p77s6"] Sep 10 02:47:42.815489 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.815414 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8e36f0_f1d8_40fa_a5c6_3da94dc53e46.slice/crio-3c679d8d7a8e3605938090cef3edddc12a59cacc8d84d2e325a86d6210b31d99 WatchSource:0}: Error finding container 3c679d8d7a8e3605938090cef3edddc12a59cacc8d84d2e325a86d6210b31d99: Status 404 returned error can't find the container with id 3c679d8d7a8e3605938090cef3edddc12a59cacc8d84d2e325a86d6210b31d99 Sep 10 02:47:42.834767 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:42.834696 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pw56k"] Sep 10 02:47:42.840646 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:42.840612 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0388e7f9_1f22_4244_9dfb_e6cf87020d8c.slice/crio-d2a2b8730cf39d27dff4256361ba595edbba15129298d53e693d1c9cab9de2c3 WatchSource:0}: Error finding container d2a2b8730cf39d27dff4256361ba595edbba15129298d53e693d1c9cab9de2c3: Status 404 returned error can't find the container with id d2a2b8730cf39d27dff4256361ba595edbba15129298d53e693d1c9cab9de2c3 Sep 10 02:47:43.549721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.549132 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ff84dbc68-p77s6" event={"ID":"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46","Type":"ContainerStarted","Data":"6b5417d4ceb899e71ad59be955380e4372aa63bdb909ec5b7288af0b96dbc693"} Sep 10 02:47:43.549721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.549193 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ff84dbc68-p77s6" event={"ID":"3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46","Type":"ContainerStarted","Data":"3c679d8d7a8e3605938090cef3edddc12a59cacc8d84d2e325a86d6210b31d99"} Sep 10 02:47:43.552598 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.552564 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56k" event={"ID":"0388e7f9-1f22-4244-9dfb-e6cf87020d8c","Type":"ContainerStarted","Data":"d2a2b8730cf39d27dff4256361ba595edbba15129298d53e693d1c9cab9de2c3"} Sep 10 02:47:43.556929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.556729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z98rn" event={"ID":"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0","Type":"ContainerStarted","Data":"f1881cb8e79908d3477fc22d4a51de510beee494649d56a577af4164943fa71a"} Sep 10 02:47:43.559762 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.559735 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" event={"ID":"848ff24f-562d-4a9b-be26-609c2b74a313","Type":"ContainerStarted","Data":"84cbc78eaac40b30e15be2c766540785f60448bddfb9954592c1e4a4d7e79df5"} Sep 10 02:47:43.564922 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.564894 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5896688487-gvlnt" event={"ID":"e8a2bf3b-5c37-4947-b77d-bc2747b2c224","Type":"ContainerStarted","Data":"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828"} Sep 10 02:47:43.565036 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.564942 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5896688487-gvlnt" event={"ID":"e8a2bf3b-5c37-4947-b77d-bc2747b2c224","Type":"ContainerStarted","Data":"bcd21c5ae6d1022e7aaa6a1c3f086faf9fbecca1c0ffbb3b7f93eb0ee4039b27"} Sep 10 02:47:43.565518 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.565452 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:47:43.578699 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.578644 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-ff84dbc68-p77s6" podStartSLOduration=40.578624022 podStartE2EDuration="40.578624022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:43.577911363 +0000 UTC m=+67.101768053" watchObservedRunningTime="2026-09-10 02:47:43.578624022 +0000 UTC m=+67.102480711" Sep 10 02:47:43.601244 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:43.601138 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5896688487-gvlnt" podStartSLOduration=45.601122699 podStartE2EDuration="45.601122699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:43.600826301 +0000 UTC m=+67.124682989" watchObservedRunningTime="2026-09-10 02:47:43.601122699 +0000 UTC m=+67.124979391" Sep 10 02:47:44.319440 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.318362 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:44.319440 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.318809 2583 scope.go:117] "RemoveContainer" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" Sep 10 02:47:44.319440 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:44.319026 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:44.319440 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.319093 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:47:44.515095 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.515058 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:44.518895 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.518647 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:44.573193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.573095 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z98rn" event={"ID":"403bbe5c-24b0-4de7-ac04-0c6fe85e81d0","Type":"ContainerStarted","Data":"d6f5a6c438fd4d1fba2f825b4b4cdd573f8f3ab2b3dc685de8a20c2541e6685c"} Sep 10 02:47:44.573193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.573184 2583 scope.go:117] "RemoveContainer" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" Sep 10 02:47:44.573647 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:44.573379 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:44.574132 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.574110 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:44.575868 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.575820 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ff84dbc68-p77s6" Sep 10 02:47:44.591557 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:44.591489 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z98rn" podStartSLOduration=66.499927639 podStartE2EDuration="1m7.591468605s" totalImagesPullingTime="1.091540966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:44.589987847 +0000 UTC m=+68.113844535" watchObservedRunningTime="2026-09-10 02:47:44.591468605 +0000 UTC m=+68.115325293" Sep 10 02:47:48.589413 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.589317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dkxnf" event={"ID":"52c15b82-ac94-4fae-a975-195f35311fd2","Type":"ContainerStarted","Data":"1a6c40d1ae3790f59302101eb6ebf063379a8f1318dcfae9031a269041f92b5c"} Sep 10 02:47:48.591093 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.590922 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:47:48.592615 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.592540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56k" event={"ID":"0388e7f9-1f22-4244-9dfb-e6cf87020d8c","Type":"ContainerStarted","Data":"3957265e20c8c2fb6e2c7434af3cc3a7dd9d15fb0edf7394fbfacad440225e2a"} Sep 10 02:47:48.597524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.597479 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" event={"ID":"848ff24f-562d-4a9b-be26-609c2b74a313","Type":"ContainerStarted","Data":"fd773621bfd7c27d9ee637fbbcfe996c91fbc4eff692fb7f3fdbc27ed98e87ef"} Sep 10 02:47:48.607662 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.607621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:48.609347 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.609298 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dkxnf" podStartSLOduration=65.749142612 podStartE2EDuration="1m11.609282192s" totalImagesPullingTime="5.86013958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:48.608462494 +0000 UTC m=+72.132319184" watchObservedRunningTime="2026-09-10 02:47:48.609282192 +0000 UTC m=+72.133138879" Sep 10 02:47:48.610651 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.610589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a116314-9fb9-4786-b01a-1bbb80aef26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77jcb\" (UID: \"1a116314-9fb9-4786-b01a-1bbb80aef26a\") " pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:48.627813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.626629 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-vblts" podStartSLOduration=40.056117425 podStartE2EDuration="45.626607873s" totalImagesPullingTime="5.570490448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:48.625872417 +0000 UTC m=+72.149729095" watchObservedRunningTime="2026-09-10 02:47:48.626607873 +0000 UTC m=+72.150464561" Sep 10 02:47:48.846371 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.846335 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fq77l\"" Sep 10 02:47:48.855114 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:48.855084 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77jcb" Sep 10 02:47:49.166723 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.166677 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77jcb"] Sep 10 02:47:49.169849 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:49.169827 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a116314_9fb9_4786_b01a_1bbb80aef26a.slice/crio-133fefdcc3f6184e8a7ee3d229d61d65b54fd7efbaeddf9d1848e5c27324752e WatchSource:0}: Error finding container 133fefdcc3f6184e8a7ee3d229d61d65b54fd7efbaeddf9d1848e5c27324752e: Status 404 returned error can't find the container with id 133fefdcc3f6184e8a7ee3d229d61d65b54fd7efbaeddf9d1848e5c27324752e Sep 10 02:47:49.607178 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.607139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56k" event={"ID":"0388e7f9-1f22-4244-9dfb-e6cf87020d8c","Type":"ContainerStarted","Data":"f8e1843c4c5f6054ee47e6ad0f2595638990c4e857974d153a833e68f33be566"} Sep 10 02:47:49.607647 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.607302 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:49.608896 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.608871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vbpc" event={"ID":"307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff","Type":"ContainerStarted","Data":"7b134ac70eba0cdf29beb6981d44a20ff6076d39882ab1a1790bc5775bedc825"} Sep 10 02:47:49.610501 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.610478 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77jcb" event={"ID":"1a116314-9fb9-4786-b01a-1bbb80aef26a","Type":"ContainerStarted","Data":"3a22f30e5c44491e0a69bfbf17bc5f5cf303efb6f3de6f404150d279ba29428e"} Sep 10 02:47:49.610585 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.610531 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77jcb" event={"ID":"1a116314-9fb9-4786-b01a-1bbb80aef26a","Type":"ContainerStarted","Data":"133fefdcc3f6184e8a7ee3d229d61d65b54fd7efbaeddf9d1848e5c27324752e"} Sep 10 02:47:49.634660 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:49.634609 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pw56k" podStartSLOduration=34.113950379 podStartE2EDuration="39.634594134s" totalImagesPullingTime="5.520643755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:49.633414288 +0000 UTC m=+73.157270976" watchObservedRunningTime="2026-09-10 02:47:49.634594134 +0000 UTC m=+73.158450844" Sep 10 02:47:50.616445 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:50.616403 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77jcb" event={"ID":"1a116314-9fb9-4786-b01a-1bbb80aef26a","Type":"ContainerStarted","Data":"2b4b727a89df08871743b0d2eebad46016f73f060f1ebc34bad47056f33d7f69"} Sep 10 02:47:52.097798 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.097735 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7vbpc" podStartSLOduration=68.907422203 podStartE2EDuration="1m15.097708141s" totalImagesPullingTime="6.190285938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:49.663942447 +0000 UTC m=+73.187799135" watchObservedRunningTime="2026-09-10 02:47:52.097708141 +0000 UTC m=+75.621564828" Sep 10 02:47:52.098701 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.098680 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-m2p7g"] Sep 10 02:47:52.121745 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.121706 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-m2p7g"] Sep 10 02:47:52.121932 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.121840 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.124216 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.124189 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 02:47:52.124354 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.124261 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 02:47:52.125119 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.125104 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 02:47:52.125187 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.125143 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-998h6\"" Sep 10 02:47:52.236115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.236077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.236115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.236119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzl59\" (UniqueName: \"kubernetes.io/projected/5f19ee98-0a8e-44db-bcc4-79395a654669-kube-api-access-mzl59\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.236331 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.236148 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f19ee98-0a8e-44db-bcc4-79395a654669-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.236331 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.236235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.337246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.337201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.337246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.337246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzl59\" (UniqueName: \"kubernetes.io/projected/5f19ee98-0a8e-44db-bcc4-79395a654669-kube-api-access-mzl59\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.337451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.337267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f19ee98-0a8e-44db-bcc4-79395a654669-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.337451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.337298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.338193 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.338165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f19ee98-0a8e-44db-bcc4-79395a654669-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.339698 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.339678 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.339801 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.339776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f19ee98-0a8e-44db-bcc4-79395a654669-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.345530 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.345502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzl59\" (UniqueName: \"kubernetes.io/projected/5f19ee98-0a8e-44db-bcc4-79395a654669-kube-api-access-mzl59\") pod \"prometheus-operator-6f4946db56-m2p7g\" (UID: \"5f19ee98-0a8e-44db-bcc4-79395a654669\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.430616 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.430587 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" Sep 10 02:47:52.625423 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.625336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77jcb" event={"ID":"1a116314-9fb9-4786-b01a-1bbb80aef26a","Type":"ContainerStarted","Data":"5a28a0abb37e975ac486d13dd565213bf4774f8f5c2290b1eb33a784e6f02bce"} Sep 10 02:47:52.644698 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.644359 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-77jcb" podStartSLOduration=33.669900022 podStartE2EDuration="36.644337636s" totalImagesPullingTime="2.974437614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:52.643540628 +0000 UTC m=+76.167397326" watchObservedRunningTime="2026-09-10 02:47:52.644337636 +0000 UTC m=+76.168194328" Sep 10 02:47:52.737311 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:52.737277 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-m2p7g"] Sep 10 02:47:52.741348 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:52.741317 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f19ee98_0a8e_44db_bcc4_79395a654669.slice/crio-3d9ad62e6e5f716e38624ea0be84c34a0553b2954fdb4acfcca0244daabc5a6f WatchSource:0}: Error finding container 3d9ad62e6e5f716e38624ea0be84c34a0553b2954fdb4acfcca0244daabc5a6f: Status 404 returned error can't find the container with id 3d9ad62e6e5f716e38624ea0be84c34a0553b2954fdb4acfcca0244daabc5a6f Sep 10 02:47:53.630615 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:53.630578 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" event={"ID":"5f19ee98-0a8e-44db-bcc4-79395a654669","Type":"ContainerStarted","Data":"3d9ad62e6e5f716e38624ea0be84c34a0553b2954fdb4acfcca0244daabc5a6f"} Sep 10 02:47:54.636155 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:54.636115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" event={"ID":"5f19ee98-0a8e-44db-bcc4-79395a654669","Type":"ContainerStarted","Data":"e476ba2d66655392098433552926eb3f6fc1d3922b1e455fbdea52106961b284"} Sep 10 02:47:55.642195 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:55.642157 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" event={"ID":"5f19ee98-0a8e-44db-bcc4-79395a654669","Type":"ContainerStarted","Data":"9b3d995083929fc34bb690d56f4455c8c0bdbaba6131dd7a8c75865c8044762f"} Sep 10 02:47:55.660539 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:55.660478 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-m2p7g" podStartSLOduration=1.990807696 podStartE2EDuration="3.660462993s" totalImagesPullingTime="1.669655297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:55.659406094 +0000 UTC m=+79.183262781" watchObservedRunningTime="2026-09-10 02:47:55.660462993 +0000 UTC m=+79.184319679" Sep 10 02:47:57.486925 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.486894 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvh29"] Sep 10 02:47:57.492633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.492605 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.494423 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.494397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 02:47:57.494645 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.494533 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ltsg\"" Sep 10 02:47:57.494645 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.494584 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 02:47:57.494746 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.494738 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 02:47:57.576796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdr4\" (UniqueName: \"kubernetes.io/projected/9ed454f6-6806-4922-a351-f21016e9d5d5-kube-api-access-hxdr4\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.576796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576802 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-sys\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-textfile\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576928 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-metrics-client-ca\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-root\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.576993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-tls\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.577028 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.577009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-wtmp\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.677739 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.677695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdr4\" (UniqueName: \"kubernetes.io/projected/9ed454f6-6806-4922-a351-f21016e9d5d5-kube-api-access-hxdr4\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.677739 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.677743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-sys\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678025 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.677807 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-sys\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678025 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.677809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678025 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.677877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-textfile\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-metrics-client-ca\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-root\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678141 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-root\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-tls\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678388 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-wtmp\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678388 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-textfile\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678388 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678357 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-wtmp\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678559 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678533 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.678661 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.678643 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ed454f6-6806-4922-a351-f21016e9d5d5-metrics-client-ca\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.680494 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.680473 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-tls\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.680601 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.680537 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ed454f6-6806-4922-a351-f21016e9d5d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.686997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.686962 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdr4\" (UniqueName: \"kubernetes.io/projected/9ed454f6-6806-4922-a351-f21016e9d5d5-kube-api-access-hxdr4\") pod \"node-exporter-rvh29\" (UID: \"9ed454f6-6806-4922-a351-f21016e9d5d5\") " pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.801374 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:57.801336 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvh29" Sep 10 02:47:57.813480 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:47:57.813439 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed454f6_6806_4922_a351_f21016e9d5d5.slice/crio-03c7a1b2eddefdc4edd7360e087a716263e3290b110be18774afaefcc2f87759 WatchSource:0}: Error finding container 03c7a1b2eddefdc4edd7360e087a716263e3290b110be18774afaefcc2f87759: Status 404 returned error can't find the container with id 03c7a1b2eddefdc4edd7360e087a716263e3290b110be18774afaefcc2f87759 Sep 10 02:47:58.655549 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:58.655502 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvh29" event={"ID":"9ed454f6-6806-4922-a351-f21016e9d5d5","Type":"ContainerStarted","Data":"03c7a1b2eddefdc4edd7360e087a716263e3290b110be18774afaefcc2f87759"} Sep 10 02:47:58.994545 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:58.994212 2583 scope.go:117] "RemoveContainer" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" Sep 10 02:47:58.994545 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:47:58.994425 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vtrdr_openshift-console-operator(be0e6efc-131e-4866-851c-27b681b19cfd)\"" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podUID="be0e6efc-131e-4866-851c-27b681b19cfd" Sep 10 02:47:59.619531 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:59.619497 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pw56k" Sep 10 02:47:59.662313 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:59.662281 2583 generic.go:356] "Generic (PLEG): container finished" podID="9ed454f6-6806-4922-a351-f21016e9d5d5" containerID="0cd51a387ccf72555ef9dce835824090b698a84f1a9d6612445ae9cfcb5c6487" exitCode=0 Sep 10 02:47:59.662665 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:47:59.662361 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvh29" event={"ID":"9ed454f6-6806-4922-a351-f21016e9d5d5","Type":"ContainerDied","Data":"0cd51a387ccf72555ef9dce835824090b698a84f1a9d6612445ae9cfcb5c6487"} Sep 10 02:48:00.668368 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:00.668332 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvh29" event={"ID":"9ed454f6-6806-4922-a351-f21016e9d5d5","Type":"ContainerStarted","Data":"5ffeae1a4fc90e4b24f8ec1ba59d96b5bc7e44f0b7707fbe82c35647d59e1ae3"} Sep 10 02:48:00.668808 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:00.668391 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvh29" event={"ID":"9ed454f6-6806-4922-a351-f21016e9d5d5","Type":"ContainerStarted","Data":"e1ccfdfbce2035faee69167841fc36c02545b00791fedac825432d501b185737"} Sep 10 02:48:00.694323 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:00.694269 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvh29" podStartSLOduration=2.486203408 podStartE2EDuration="3.694253746s" totalImagesPullingTime="1.142688024s" totalInitContainerRuntime="65.362314ms" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:00.693409433 +0000 UTC m=+84.217266121" watchObservedRunningTime="2026-09-10 02:48:00.694253746 +0000 UTC m=+84.218110433" Sep 10 02:48:01.843039 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.843002 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6498c9dcbf-rvp4x"] Sep 10 02:48:01.846398 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.846376 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:01.848641 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.848623 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 02:48:01.848767 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.848713 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 02:48:01.848824 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.848782 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3eh5g1hsldh5f\"" Sep 10 02:48:01.849382 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.849363 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 02:48:01.849486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.849365 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6hchb\"" Sep 10 02:48:01.849486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.849408 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 02:48:01.858504 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:01.858480 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6498c9dcbf-rvp4x"] Sep 10 02:48:02.012954 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.012909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-client-certs\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013161 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-audit-log\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013161 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-tls\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013253 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013253 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-client-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013253 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxq7\" (UniqueName: \"kubernetes.io/projected/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-kube-api-access-8dxq7\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.013370 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.013263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-metrics-server-audit-profiles\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114033 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.113944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-metrics-server-audit-profiles\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114033 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.113981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-client-certs\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-audit-log\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-tls\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-client-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxq7\" (UniqueName: \"kubernetes.io/projected/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-kube-api-access-8dxq7\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114602 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114573 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-audit-log\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.114948 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.114922 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.115166 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.115145 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-metrics-server-audit-profiles\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.116614 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.116594 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-client-certs\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.116696 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.116654 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-client-ca-bundle\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.116814 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.116790 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-secret-metrics-server-tls\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.123969 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.123943 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxq7\" (UniqueName: \"kubernetes.io/projected/ba5cc5c6-7638-4f79-943d-0eeebf58e34e-kube-api-access-8dxq7\") pod \"metrics-server-6498c9dcbf-rvp4x\" (UID: \"ba5cc5c6-7638-4f79-943d-0eeebf58e34e\") " pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.155399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.155355 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:02.233960 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.232039 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8"] Sep 10 02:48:02.237246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.237224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.240123 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.240088 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g6gb5\"" Sep 10 02:48:02.240619 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.240594 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 02:48:02.245853 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.245824 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8"] Sep 10 02:48:02.416551 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.416454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aca351-0593-4dee-bf0f-975c649e2153-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.416551 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.416541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptwz\" (UniqueName: \"kubernetes.io/projected/f8aca351-0593-4dee-bf0f-975c649e2153-kube-api-access-8ptwz\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.443732 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.443562 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6498c9dcbf-rvp4x"] Sep 10 02:48:02.446441 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:02.446412 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5cc5c6_7638_4f79_943d_0eeebf58e34e.slice/crio-7ba07a35eaade0cee3d57856d14269eac316ee84e097e3cf2ebe0b46a675c248 WatchSource:0}: Error finding container 7ba07a35eaade0cee3d57856d14269eac316ee84e097e3cf2ebe0b46a675c248: Status 404 returned error can't find the container with id 7ba07a35eaade0cee3d57856d14269eac316ee84e097e3cf2ebe0b46a675c248 Sep 10 02:48:02.517460 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.517429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptwz\" (UniqueName: \"kubernetes.io/projected/f8aca351-0593-4dee-bf0f-975c649e2153-kube-api-access-8ptwz\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.517621 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.517478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aca351-0593-4dee-bf0f-975c649e2153-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.519796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.519776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aca351-0593-4dee-bf0f-975c649e2153-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.527643 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.527619 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptwz\" (UniqueName: \"kubernetes.io/projected/f8aca351-0593-4dee-bf0f-975c649e2153-kube-api-access-8ptwz\") pod \"monitoring-plugin-85bbbbbf4-t46f8\" (UID: \"f8aca351-0593-4dee-bf0f-975c649e2153\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.549654 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.549623 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:02.670359 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.670279 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz"] Sep 10 02:48:02.674285 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.674261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.676681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676645 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 02:48:02.676681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676645 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 02:48:02.676929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676652 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 02:48:02.676929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676652 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c7jld\"" Sep 10 02:48:02.676929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676797 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 02:48:02.676929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.676652 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 02:48:02.677643 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.677616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" event={"ID":"ba5cc5c6-7638-4f79-943d-0eeebf58e34e","Type":"ContainerStarted","Data":"7ba07a35eaade0cee3d57856d14269eac316ee84e097e3cf2ebe0b46a675c248"} Sep 10 02:48:02.682283 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.682262 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 02:48:02.691524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.691503 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz"] Sep 10 02:48:02.819570 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-federate-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6sk\" (UniqueName: \"kubernetes.io/projected/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-kube-api-access-9q6sk\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-serving-certs-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819771 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819802 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.819883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.819825 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-metrics-client-ca\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.828257 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.828233 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8"] Sep 10 02:48:02.831709 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:02.831683 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aca351_0593_4dee_bf0f_975c649e2153.slice/crio-bc90e87147412a787c7c9a962dae7cce08ededdc6faa2d6bf638c22c4a54df6e WatchSource:0}: Error finding container bc90e87147412a787c7c9a962dae7cce08ededdc6faa2d6bf638c22c4a54df6e: Status 404 returned error can't find the container with id bc90e87147412a787c7c9a962dae7cce08ededdc6faa2d6bf638c22c4a54df6e Sep 10 02:48:02.921003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.920915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-federate-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.920980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6sk\" (UniqueName: \"kubernetes.io/projected/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-kube-api-access-9q6sk\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-serving-certs-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921094 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.921453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.921185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-metrics-client-ca\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.922140 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.922088 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-metrics-client-ca\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.922444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.922377 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.922444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.922413 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-serving-certs-ca-bundle\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.924076 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.924050 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-federate-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.924529 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.924502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.924691 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.924672 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-telemeter-client-tls\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.924786 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.924763 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.932513 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.932489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6sk\" (UniqueName: \"kubernetes.io/projected/a2a4c2f3-f2d1-490d-9894-31eaaf654ec7-kube-api-access-9q6sk\") pod \"telemeter-client-56c5cd9c7c-tncxz\" (UID: \"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7\") " pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:02.984474 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:02.984440 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" Sep 10 02:48:03.278239 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:03.278198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz"] Sep 10 02:48:03.281875 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:03.281828 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a4c2f3_f2d1_490d_9894_31eaaf654ec7.slice/crio-494b4ac319f4e52e7d6906d63b2e2c6d36a0ddd99cd5e1502fc1f2fe1249082d WatchSource:0}: Error finding container 494b4ac319f4e52e7d6906d63b2e2c6d36a0ddd99cd5e1502fc1f2fe1249082d: Status 404 returned error can't find the container with id 494b4ac319f4e52e7d6906d63b2e2c6d36a0ddd99cd5e1502fc1f2fe1249082d Sep 10 02:48:03.684426 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:03.684317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" event={"ID":"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7","Type":"ContainerStarted","Data":"494b4ac319f4e52e7d6906d63b2e2c6d36a0ddd99cd5e1502fc1f2fe1249082d"} Sep 10 02:48:03.685779 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:03.685736 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" event={"ID":"f8aca351-0593-4dee-bf0f-975c649e2153","Type":"ContainerStarted","Data":"bc90e87147412a787c7c9a962dae7cce08ededdc6faa2d6bf638c22c4a54df6e"} Sep 10 02:48:04.693716 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:04.693651 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" event={"ID":"ba5cc5c6-7638-4f79-943d-0eeebf58e34e","Type":"ContainerStarted","Data":"43d41a7130850c69986c139714ab502f434f12d2ed84742aacf47651c1c99d18"} Sep 10 02:48:04.713143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:04.713090 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" podStartSLOduration=2.408027454 podStartE2EDuration="3.713070577s" totalImagesPullingTime="1.305043123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:04.712049131 +0000 UTC m=+88.235905819" watchObservedRunningTime="2026-09-10 02:48:04.713070577 +0000 UTC m=+88.236927264" Sep 10 02:48:05.581694 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.581612 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:48:05.688917 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.688875 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:48:05.699363 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.699329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" event={"ID":"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7","Type":"ContainerStarted","Data":"aea9cc2385848ef55bb97890c56917cb7534d5421ce704f005d1d790ca0709e7"} Sep 10 02:48:05.701201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.701161 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" event={"ID":"f8aca351-0593-4dee-bf0f-975c649e2153","Type":"ContainerStarted","Data":"477108eea2b8dc8703d4d5f44452cb3b8e6e597ac1ccf002e6c404239af8258c"} Sep 10 02:48:05.701347 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.701238 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:05.706380 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.706358 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" Sep 10 02:48:05.717834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:05.717785 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t46f8" podStartSLOduration=1.7696101629999998 podStartE2EDuration="3.717768076s" totalImagesPullingTime="1.948157913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:05.717309575 +0000 UTC m=+89.241166264" watchObservedRunningTime="2026-09-10 02:48:05.717768076 +0000 UTC m=+89.241624763" Sep 10 02:48:06.706733 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:06.706625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" event={"ID":"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7","Type":"ContainerStarted","Data":"8cb137e9317f4508b7ff224626e427545a0e09e0cde3956cfa8ca7f2b624b3d5"} Sep 10 02:48:06.706733 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:06.706716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" event={"ID":"a2a4c2f3-f2d1-490d-9894-31eaaf654ec7","Type":"ContainerStarted","Data":"9661744e35742dffa09248e6a89a618a4789cc779dbacccfd46fe2fd9ab37b38"} Sep 10 02:48:06.728168 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:06.728120 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56c5cd9c7c-tncxz" podStartSLOduration=1.6793186530000002 podStartE2EDuration="4.72810566s" totalImagesPullingTime="3.048787007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:06.72757535 +0000 UTC m=+90.251432037" watchObservedRunningTime="2026-09-10 02:48:06.72810566 +0000 UTC m=+90.251962347" Sep 10 02:48:10.994427 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:10.994384 2583 scope.go:117] "RemoveContainer" containerID="6d562d7b12a630a3346f843697b1af0a4420ab4a06c6962c8c8017c8c893c8b4" Sep 10 02:48:11.730494 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.730466 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:48:11.730659 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.730552 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" event={"ID":"be0e6efc-131e-4866-851c-27b681b19cfd","Type":"ContainerStarted","Data":"c8306ba0228915f5ce50bcd8a240a0212f7d4798f7cad1ab0fe3b52d0baad1e3"} Sep 10 02:48:11.730842 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.730812 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:48:11.735875 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.735833 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" Sep 10 02:48:11.753196 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.753135 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-vtrdr" podStartSLOduration=54.477304918 podStartE2EDuration="58.753112986s" totalImagesPullingTime="4.275808068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:11.751670259 +0000 UTC m=+95.275526947" watchObservedRunningTime="2026-09-10 02:48:11.753112986 +0000 UTC m=+95.276969674" Sep 10 02:48:11.805758 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.805728 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-6mqnf"] Sep 10 02:48:11.809041 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.809018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:11.811210 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.811188 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-j27nk\"" Sep 10 02:48:11.811533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.811518 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 02:48:11.811647 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.811620 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 02:48:11.822103 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.822078 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6mqnf"] Sep 10 02:48:11.902211 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:11.902172 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8shx\" (UniqueName: \"kubernetes.io/projected/80363e78-04f3-4944-b63f-83ff136f2ed9-kube-api-access-w8shx\") pod \"downloads-85d98c998f-6mqnf\" (UID: \"80363e78-04f3-4944-b63f-83ff136f2ed9\") " pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:12.003575 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:12.003542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8shx\" (UniqueName: \"kubernetes.io/projected/80363e78-04f3-4944-b63f-83ff136f2ed9-kube-api-access-w8shx\") pod \"downloads-85d98c998f-6mqnf\" (UID: \"80363e78-04f3-4944-b63f-83ff136f2ed9\") " pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:12.011623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:12.011595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8shx\" (UniqueName: \"kubernetes.io/projected/80363e78-04f3-4944-b63f-83ff136f2ed9-kube-api-access-w8shx\") pod \"downloads-85d98c998f-6mqnf\" (UID: \"80363e78-04f3-4944-b63f-83ff136f2ed9\") " pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:12.119185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:12.119148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:12.393512 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:12.393438 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6mqnf"] Sep 10 02:48:12.397085 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:12.397057 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80363e78_04f3_4944_b63f_83ff136f2ed9.slice/crio-95b58961555550e5541aea9e761f00d2f3c4251c48a271f388de196ee645f93a WatchSource:0}: Error finding container 95b58961555550e5541aea9e761f00d2f3c4251c48a271f388de196ee645f93a: Status 404 returned error can't find the container with id 95b58961555550e5541aea9e761f00d2f3c4251c48a271f388de196ee645f93a Sep 10 02:48:12.737596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:12.737515 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6mqnf" event={"ID":"80363e78-04f3-4944-b63f-83ff136f2ed9","Type":"ContainerStarted","Data":"95b58961555550e5541aea9e761f00d2f3c4251c48a271f388de196ee645f93a"} Sep 10 02:48:18.184485 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.184437 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:48:18.208981 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.208949 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:48:18.209150 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.209045 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.211229 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.211206 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 02:48:18.211581 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.211561 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rq5sw\"" Sep 10 02:48:18.212174 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.211996 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 02:48:18.212174 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.212065 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 02:48:18.212174 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.212068 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 02:48:18.212462 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.212421 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 02:48:18.263147 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.263319 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.263319 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.263465 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pzs\" (UniqueName: \"kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.263465 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.263465 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.263412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364693 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pzs\" (UniqueName: \"kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.364915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.364875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.365697 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.365500 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.365697 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.365509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.365883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.365828 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.367716 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.367695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.367842 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.367748 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.373103 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.373072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pzs\" (UniqueName: \"kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs\") pod \"console-7cccc64d5c-j5d7m\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.521337 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.521292 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:18.819586 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:18.819500 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:48:18.823471 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:18.823429 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf33562b_36ab_4f2f_b7d1_1545266b2e97.slice/crio-f2c3c267007fea979b609fea7ea0d80452999bdf3a2ea6c13e53d83b126622e6 WatchSource:0}: Error finding container f2c3c267007fea979b609fea7ea0d80452999bdf3a2ea6c13e53d83b126622e6: Status 404 returned error can't find the container with id f2c3c267007fea979b609fea7ea0d80452999bdf3a2ea6c13e53d83b126622e6 Sep 10 02:48:19.776183 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:19.776142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cccc64d5c-j5d7m" event={"ID":"af33562b-36ab-4f2f-b7d1-1545266b2e97","Type":"ContainerStarted","Data":"f2c3c267007fea979b609fea7ea0d80452999bdf3a2ea6c13e53d83b126622e6"} Sep 10 02:48:20.619290 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:20.619256 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dkxnf" Sep 10 02:48:22.156517 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:22.156479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:22.157052 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:22.156563 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:27.902172 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.902138 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:48:27.905571 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.905548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.914391 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.914320 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:48:27.914533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.914493 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 02:48:27.960376 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960376 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960616 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960616 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960616 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960616 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960597 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhm6d\" (UniqueName: \"kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:27.960772 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:27.960654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061082 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm6d\" (UniqueName: \"kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061082 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061079 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061336 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061336 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061336 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061336 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.061336 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.061234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.062119 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.062085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.062258 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.062190 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.062258 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.062235 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.062375 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.062282 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.063953 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.063929 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.064128 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.064108 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.069833 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.069806 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm6d\" (UniqueName: \"kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d\") pod \"console-76f656f477-js47t\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:28.216972 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:28.216880 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:29.168643 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.168550 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:48:29.173056 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:48:29.173013 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fc470e_09df_458b_87d5_cb008ac84621.slice/crio-857b443623f3883eda1b1299bee847fb5f4841f5263c5115ccd61f2af5f5e610 WatchSource:0}: Error finding container 857b443623f3883eda1b1299bee847fb5f4841f5263c5115ccd61f2af5f5e610: Status 404 returned error can't find the container with id 857b443623f3883eda1b1299bee847fb5f4841f5263c5115ccd61f2af5f5e610 Sep 10 02:48:29.822210 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.822159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6mqnf" event={"ID":"80363e78-04f3-4944-b63f-83ff136f2ed9","Type":"ContainerStarted","Data":"7a81d51ab890973358fcf65614b25fa1fc11d8d7a699e31372dfccd481cddffb"} Sep 10 02:48:29.822382 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.822356 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:29.824049 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.824015 2583 generic.go:356] "Generic (PLEG): container finished" podID="26fd4e22-d22f-429e-b429-a4c64cdc789d" containerID="d174f50b5843ea39b578007772e4767a12f74eed03c05cf9098744a6b78fc83d" exitCode=0 Sep 10 02:48:29.824171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.824043 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" event={"ID":"26fd4e22-d22f-429e-b429-a4c64cdc789d","Type":"ContainerDied","Data":"d174f50b5843ea39b578007772e4767a12f74eed03c05cf9098744a6b78fc83d"} Sep 10 02:48:29.824408 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.824388 2583 scope.go:117] "RemoveContainer" containerID="d174f50b5843ea39b578007772e4767a12f74eed03c05cf9098744a6b78fc83d" Sep 10 02:48:29.826203 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.826148 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cccc64d5c-j5d7m" event={"ID":"af33562b-36ab-4f2f-b7d1-1545266b2e97","Type":"ContainerStarted","Data":"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921"} Sep 10 02:48:29.828106 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.828080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f656f477-js47t" event={"ID":"10fc470e-09df-458b-87d5-cb008ac84621","Type":"ContainerStarted","Data":"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945"} Sep 10 02:48:29.828187 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.828122 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f656f477-js47t" event={"ID":"10fc470e-09df-458b-87d5-cb008ac84621","Type":"ContainerStarted","Data":"857b443623f3883eda1b1299bee847fb5f4841f5263c5115ccd61f2af5f5e610"} Sep 10 02:48:29.837283 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.837256 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-6mqnf" Sep 10 02:48:29.842667 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.842605 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-6mqnf" podStartSLOduration=2.293254979 podStartE2EDuration="18.842585583s" totalImagesPullingTime="16.549330604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:29.841847924 +0000 UTC m=+113.365704622" watchObservedRunningTime="2026-09-10 02:48:29.842585583 +0000 UTC m=+113.366442271" Sep 10 02:48:29.887104 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.887050 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cccc64d5c-j5d7m" podStartSLOduration=1.816569257 podStartE2EDuration="11.887031364s" totalImagesPullingTime="10.070462107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:29.884904425 +0000 UTC m=+113.408761112" watchObservedRunningTime="2026-09-10 02:48:29.887031364 +0000 UTC m=+113.410888051" Sep 10 02:48:29.956049 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:29.955979 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76f656f477-js47t" podStartSLOduration=2.955957219 podStartE2EDuration="2.955957219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:29.953250445 +0000 UTC m=+113.477107131" watchObservedRunningTime="2026-09-10 02:48:29.955957219 +0000 UTC m=+113.479813907" Sep 10 02:48:30.721228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:30.721178 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5896688487-gvlnt" podUID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" containerName="registry" containerID="cri-o://421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828" gracePeriod=30 Sep 10 02:48:30.835792 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:30.835748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2wn9k" event={"ID":"26fd4e22-d22f-429e-b429-a4c64cdc789d","Type":"ContainerStarted","Data":"fb40af6fd934ad5e2943272116c0f1cc57ed5f5000cb5ad5a8c9da089d10427a"} Sep 10 02:48:31.041945 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.041906 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:48:31.092958 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.092920 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093159 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.092992 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093159 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093050 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093159 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093099 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj44x\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093159 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093134 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093372 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093163 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093372 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093190 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093372 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093258 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates\") pod \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\" (UID: \"e8a2bf3b-5c37-4947-b77d-bc2747b2c224\") " Sep 10 02:48:31.093877 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093681 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:48:31.093996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.093897 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:48:31.098190 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.098120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x" (OuterVolumeSpecName: "kube-api-access-kj44x") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "kube-api-access-kj44x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:48:31.098323 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.098197 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:48:31.098389 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.098369 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:48:31.098987 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.098957 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:48:31.100729 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.100702 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:48:31.106356 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.106320 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e8a2bf3b-5c37-4947-b77d-bc2747b2c224" (UID: "e8a2bf3b-5c37-4947-b77d-bc2747b2c224"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:48:31.194096 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194056 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-certificates\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194096 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194092 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-registry-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194108 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-image-registry-private-configuration\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194122 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-bound-sa-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194133 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj44x\" (UniqueName: \"kubernetes.io/projected/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-kube-api-access-kj44x\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194147 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-installation-pull-secrets\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194157 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-trusted-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.194307 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.194168 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8a2bf3b-5c37-4947-b77d-bc2747b2c224-ca-trust-extracted\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:48:31.841689 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.841656 2583 generic.go:356] "Generic (PLEG): container finished" podID="419faac4-860c-472c-b9c7-1fb983cc3085" containerID="4a0dfd3d178ea0d454fabad801ca6dacde0c8cd10dc25b8da3140f5dd7788555" exitCode=0 Sep 10 02:48:31.842175 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.841732 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" event={"ID":"419faac4-860c-472c-b9c7-1fb983cc3085","Type":"ContainerDied","Data":"4a0dfd3d178ea0d454fabad801ca6dacde0c8cd10dc25b8da3140f5dd7788555"} Sep 10 02:48:31.842252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.842233 2583 scope.go:117] "RemoveContainer" containerID="4a0dfd3d178ea0d454fabad801ca6dacde0c8cd10dc25b8da3140f5dd7788555" Sep 10 02:48:31.843764 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.843738 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" containerID="421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828" exitCode=0 Sep 10 02:48:31.843919 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.843790 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5896688487-gvlnt" Sep 10 02:48:31.843919 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.843809 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5896688487-gvlnt" event={"ID":"e8a2bf3b-5c37-4947-b77d-bc2747b2c224","Type":"ContainerDied","Data":"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828"} Sep 10 02:48:31.843919 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.843902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5896688487-gvlnt" event={"ID":"e8a2bf3b-5c37-4947-b77d-bc2747b2c224","Type":"ContainerDied","Data":"bcd21c5ae6d1022e7aaa6a1c3f086faf9fbecca1c0ffbb3b7f93eb0ee4039b27"} Sep 10 02:48:31.844081 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.843934 2583 scope.go:117] "RemoveContainer" containerID="421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828" Sep 10 02:48:31.861402 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.861377 2583 scope.go:117] "RemoveContainer" containerID="421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828" Sep 10 02:48:31.861810 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:48:31.861783 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828\": container with ID starting with 421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828 not found: ID does not exist" containerID="421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828" Sep 10 02:48:31.861912 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.861823 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828"} err="failed to get container status \"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828\": rpc error: code = NotFound desc = could not find container \"421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828\": container with ID starting with 421f59455624d09f7cd4abe1abc56ac6f9a75b57106c3be0c3a5c0825549d828 not found: ID does not exist" Sep 10 02:48:31.883537 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.883503 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:48:31.891782 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:31.891734 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5896688487-gvlnt"] Sep 10 02:48:32.850494 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:32.850457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kj7w5" event={"ID":"419faac4-860c-472c-b9c7-1fb983cc3085","Type":"ContainerStarted","Data":"7a7daa5e9fe93985fd77895361e68b54051317d8b66f43f4248568d09996c7cb"} Sep 10 02:48:33.000262 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:33.000223 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" path="/var/lib/kubelet/pods/e8a2bf3b-5c37-4947-b77d-bc2747b2c224/volumes" Sep 10 02:48:38.217962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.217920 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:38.218467 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.217978 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:38.222796 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.222771 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:38.521731 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.521693 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:38.521957 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.521758 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:38.526705 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.526679 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:38.884026 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.883936 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:48:38.884455 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.884430 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:48:38.946249 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:38.946197 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:48:42.162583 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:42.162547 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:42.166658 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:42.166627 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6498c9dcbf-rvp4x" Sep 10 02:48:46.923575 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:46.923536 2583 generic.go:356] "Generic (PLEG): container finished" podID="ec1c1665-a796-4a58-8798-14a539c3e8fd" containerID="74e1726db8c130a8c41e3362687389e7caa7d1ffcbae000373b2a3bfca80547f" exitCode=0 Sep 10 02:48:46.923575 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:46.923577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" event={"ID":"ec1c1665-a796-4a58-8798-14a539c3e8fd","Type":"ContainerDied","Data":"74e1726db8c130a8c41e3362687389e7caa7d1ffcbae000373b2a3bfca80547f"} Sep 10 02:48:46.924099 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:46.923969 2583 scope.go:117] "RemoveContainer" containerID="74e1726db8c130a8c41e3362687389e7caa7d1ffcbae000373b2a3bfca80547f" Sep 10 02:48:47.929927 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:48:47.929888 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-sdjcs" event={"ID":"ec1c1665-a796-4a58-8798-14a539c3e8fd","Type":"ContainerStarted","Data":"66b2c194dd8bbfff6ef9ab3747931ead1631b127c18a59ff0f4ddb96cd0ababe"} Sep 10 02:49:05.909439 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:05.909371 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cccc64d5c-j5d7m" podUID="af33562b-36ab-4f2f-b7d1-1545266b2e97" containerName="console" containerID="cri-o://e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921" gracePeriod=15 Sep 10 02:49:06.199162 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.199123 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:49:06.297345 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297306 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297367 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297438 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297454 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297491 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9pzs\" (UniqueName: \"kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs\") pod \"af33562b-36ab-4f2f-b7d1-1545266b2e97\" (UID: \"af33562b-36ab-4f2f-b7d1-1545266b2e97\") " Sep 10 02:49:06.297883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297826 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config" (OuterVolumeSpecName: "console-config") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:49:06.297980 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297954 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:49:06.298042 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.297950 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca" (OuterVolumeSpecName: "service-ca") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:49:06.299810 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.299782 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs" (OuterVolumeSpecName: "kube-api-access-l9pzs") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "kube-api-access-l9pzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:49:06.300077 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.300056 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:49:06.300152 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.300073 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af33562b-36ab-4f2f-b7d1-1545266b2e97" (UID: "af33562b-36ab-4f2f-b7d1-1545266b2e97"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:49:06.398741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398699 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pzs\" (UniqueName: \"kubernetes.io/projected/af33562b-36ab-4f2f-b7d1-1545266b2e97-kube-api-access-l9pzs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:06.398741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398732 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:06.398741 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398747 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:06.399015 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398759 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af33562b-36ab-4f2f-b7d1-1545266b2e97-console-oauth-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:06.399015 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398771 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-oauth-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:06.399015 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:06.398786 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af33562b-36ab-4f2f-b7d1-1545266b2e97-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:49:07.006655 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.006623 2583 generic.go:356] "Generic (PLEG): container finished" podID="af33562b-36ab-4f2f-b7d1-1545266b2e97" containerID="e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921" exitCode=0 Sep 10 02:49:07.007117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.006688 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cccc64d5c-j5d7m" Sep 10 02:49:07.007117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.006720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cccc64d5c-j5d7m" event={"ID":"af33562b-36ab-4f2f-b7d1-1545266b2e97","Type":"ContainerDied","Data":"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921"} Sep 10 02:49:07.007117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.006768 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cccc64d5c-j5d7m" event={"ID":"af33562b-36ab-4f2f-b7d1-1545266b2e97","Type":"ContainerDied","Data":"f2c3c267007fea979b609fea7ea0d80452999bdf3a2ea6c13e53d83b126622e6"} Sep 10 02:49:07.007117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.006792 2583 scope.go:117] "RemoveContainer" containerID="e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921" Sep 10 02:49:07.017130 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.017044 2583 scope.go:117] "RemoveContainer" containerID="e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921" Sep 10 02:49:07.017419 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:49:07.017398 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921\": container with ID starting with e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921 not found: ID does not exist" containerID="e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921" Sep 10 02:49:07.017486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.017426 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921"} err="failed to get container status \"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921\": rpc error: code = NotFound desc = could not find container \"e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921\": container with ID starting with e139a968bb3e953adc4812d07e8a714387d3e70664853e86d13b92b99123f921 not found: ID does not exist" Sep 10 02:49:07.023466 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.023444 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:49:07.026596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:07.026571 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cccc64d5c-j5d7m"] Sep 10 02:49:08.999351 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:08.999312 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af33562b-36ab-4f2f-b7d1-1545266b2e97" path="/var/lib/kubelet/pods/af33562b-36ab-4f2f-b7d1-1545266b2e97/volumes" Sep 10 02:49:24.560997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.560963 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:49:24.561997 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.561970 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af33562b-36ab-4f2f-b7d1-1545266b2e97" containerName="console" Sep 10 02:49:24.562139 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.562127 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af33562b-36ab-4f2f-b7d1-1545266b2e97" containerName="console" Sep 10 02:49:24.562252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.562241 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" containerName="registry" Sep 10 02:49:24.562330 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.562320 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" containerName="registry" Sep 10 02:49:24.562496 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.562481 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a2bf3b-5c37-4947-b77d-bc2747b2c224" containerName="registry" Sep 10 02:49:24.562596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.562584 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af33562b-36ab-4f2f-b7d1-1545266b2e97" containerName="console" Sep 10 02:49:24.566903 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.566882 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.575254 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.575229 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:49:24.660720 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.660677 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.660720 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.660730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.661035 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.660753 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.661035 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.660890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.661035 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.660958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplpw\" (UniqueName: \"kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.661035 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.661018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.661190 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.661041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.761961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762003 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplpw\" (UniqueName: \"kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762854 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762817 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762826 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762977 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.762996 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.762985 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.764562 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.764532 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.764670 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.764651 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.771633 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.771608 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplpw\" (UniqueName: \"kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw\") pod \"console-b5fb5ccb5-z2hll\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:24.877640 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:24.877551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:25.150651 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:25.150625 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:49:25.153056 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:49:25.153028 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b20aaf1_eb66_4ac4_adc0_7c9a75e155a0.slice/crio-c90dfce05cd616140d2193a1c4b04c672fbf36aa94e90cc3ac3af0d38d29d169 WatchSource:0}: Error finding container c90dfce05cd616140d2193a1c4b04c672fbf36aa94e90cc3ac3af0d38d29d169: Status 404 returned error can't find the container with id c90dfce05cd616140d2193a1c4b04c672fbf36aa94e90cc3ac3af0d38d29d169 Sep 10 02:49:26.082232 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:26.082193 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fb5ccb5-z2hll" event={"ID":"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0","Type":"ContainerStarted","Data":"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075"} Sep 10 02:49:26.082709 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:26.082249 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fb5ccb5-z2hll" event={"ID":"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0","Type":"ContainerStarted","Data":"c90dfce05cd616140d2193a1c4b04c672fbf36aa94e90cc3ac3af0d38d29d169"} Sep 10 02:49:26.102380 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:26.102322 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5fb5ccb5-z2hll" podStartSLOduration=2.102305892 podStartE2EDuration="2.102305892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:49:26.101493843 +0000 UTC m=+169.625350533" watchObservedRunningTime="2026-09-10 02:49:26.102305892 +0000 UTC m=+169.626162578" Sep 10 02:49:34.878411 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:34.878317 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:34.878411 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:34.878385 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:34.883459 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:34.883435 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:35.122975 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:35.122943 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:49:35.169196 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:49:35.169102 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:50:00.195426 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.195358 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76f656f477-js47t" podUID="10fc470e-09df-458b-87d5-cb008ac84621" containerName="console" containerID="cri-o://2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945" gracePeriod=15 Sep 10 02:50:00.443990 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.443958 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:50:00.456889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456807 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.456889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456842 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.456889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456877 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.457115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456907 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.457115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456933 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.457115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.456954 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.457115 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.457022 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhm6d\" (UniqueName: \"kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d\") pod \"10fc470e-09df-458b-87d5-cb008ac84621\" (UID: \"10fc470e-09df-458b-87d5-cb008ac84621\") " Sep 10 02:50:00.457301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.457259 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:50:00.457406 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.457380 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca" (OuterVolumeSpecName: "service-ca") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:50:00.457449 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.457416 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config" (OuterVolumeSpecName: "console-config") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:50:00.457449 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.457430 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:50:00.459119 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.459093 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:50:00.459211 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.459186 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:50:00.459280 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.459259 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d" (OuterVolumeSpecName: "kube-api-access-mhm6d") pod "10fc470e-09df-458b-87d5-cb008ac84621" (UID: "10fc470e-09df-458b-87d5-cb008ac84621"). InnerVolumeSpecName "kube-api-access-mhm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:50:00.557594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557551 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhm6d\" (UniqueName: \"kubernetes.io/projected/10fc470e-09df-458b-87d5-cb008ac84621-kube-api-access-mhm6d\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557587 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-oauth-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557601 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-oauth-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557613 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557626 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10fc470e-09df-458b-87d5-cb008ac84621-console-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557636 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-console-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:00.557834 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:00.557647 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10fc470e-09df-458b-87d5-cb008ac84621-trusted-ca-bundle\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:01.219984 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.219949 2583 generic.go:356] "Generic (PLEG): container finished" podID="10fc470e-09df-458b-87d5-cb008ac84621" containerID="2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945" exitCode=0 Sep 10 02:50:01.220413 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.219992 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f656f477-js47t" Sep 10 02:50:01.220413 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.220023 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f656f477-js47t" event={"ID":"10fc470e-09df-458b-87d5-cb008ac84621","Type":"ContainerDied","Data":"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945"} Sep 10 02:50:01.220413 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.220076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f656f477-js47t" event={"ID":"10fc470e-09df-458b-87d5-cb008ac84621","Type":"ContainerDied","Data":"857b443623f3883eda1b1299bee847fb5f4841f5263c5115ccd61f2af5f5e610"} Sep 10 02:50:01.220413 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.220097 2583 scope.go:117] "RemoveContainer" containerID="2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945" Sep 10 02:50:01.228447 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.228428 2583 scope.go:117] "RemoveContainer" containerID="2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945" Sep 10 02:50:01.228715 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:50:01.228694 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945\": container with ID starting with 2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945 not found: ID does not exist" containerID="2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945" Sep 10 02:50:01.228775 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.228724 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945"} err="failed to get container status \"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945\": rpc error: code = NotFound desc = could not find container \"2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945\": container with ID starting with 2850dab8beef170eaf6c1ef36039676ee46af7757d91bf92beade36ba65af945 not found: ID does not exist" Sep 10 02:50:01.239888 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.239841 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:50:01.242247 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:01.242222 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f656f477-js47t"] Sep 10 02:50:02.999011 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:02.998979 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10fc470e-09df-458b-87d5-cb008ac84621" path="/var/lib/kubelet/pods/10fc470e-09df-458b-87d5-cb008ac84621/volumes" Sep 10 02:50:32.845446 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.845406 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h"] Sep 10 02:50:32.846091 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.845752 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fc470e-09df-458b-87d5-cb008ac84621" containerName="console" Sep 10 02:50:32.846091 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.845768 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fc470e-09df-458b-87d5-cb008ac84621" containerName="console" Sep 10 02:50:32.846091 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.845870 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fc470e-09df-458b-87d5-cb008ac84621" containerName="console" Sep 10 02:50:32.849049 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.849029 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:32.851536 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.851513 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllqd\"" Sep 10 02:50:32.851768 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.851748 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 02:50:32.852228 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.852211 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 02:50:32.859089 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.859062 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h"] Sep 10 02:50:32.927234 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.927191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpct\" (UniqueName: \"kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:32.927234 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.927245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:32.927494 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:32.927331 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.028194 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.028154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.028359 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.028211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.028359 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.028270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpct\" (UniqueName: \"kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.028540 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.028509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.028581 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.028548 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.036787 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.036758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpct\" (UniqueName: \"kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.159277 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.159186 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:33.432335 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:33.432141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h"] Sep 10 02:50:33.435059 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:50:33.435024 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91915c_aec9_4a18_93ca_e9b590ac9ffe.slice/crio-0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc WatchSource:0}: Error finding container 0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc: Status 404 returned error can't find the container with id 0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc Sep 10 02:50:34.351072 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:34.351027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" event={"ID":"ce91915c-aec9-4a18-93ca-e9b590ac9ffe","Type":"ContainerStarted","Data":"0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc"} Sep 10 02:50:40.380205 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:40.380171 2583 generic.go:356] "Generic (PLEG): container finished" podID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerID="92a5dd030e22bd9897a11bed989b5f50a77e6d2953c7df7e64afb59097b36b20" exitCode=0 Sep 10 02:50:40.380605 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:40.380267 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" event={"ID":"ce91915c-aec9-4a18-93ca-e9b590ac9ffe","Type":"ContainerDied","Data":"92a5dd030e22bd9897a11bed989b5f50a77e6d2953c7df7e64afb59097b36b20"} Sep 10 02:50:43.394628 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:43.394594 2583 generic.go:356] "Generic (PLEG): container finished" podID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerID="5f7e68ffb90f61e16751cf68b153b9381597ab139810dfbc82fd8cd6c093f432" exitCode=0 Sep 10 02:50:43.395063 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:43.394685 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" event={"ID":"ce91915c-aec9-4a18-93ca-e9b590ac9ffe","Type":"ContainerDied","Data":"5f7e68ffb90f61e16751cf68b153b9381597ab139810dfbc82fd8cd6c093f432"} Sep 10 02:50:50.429583 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:50.429539 2583 generic.go:356] "Generic (PLEG): container finished" podID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerID="8939a85ba831028a2a2cca79b067dc720f07a112ea075407b6ed9de965540504" exitCode=0 Sep 10 02:50:50.430043 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:50.429621 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" event={"ID":"ce91915c-aec9-4a18-93ca-e9b590ac9ffe","Type":"ContainerDied","Data":"8939a85ba831028a2a2cca79b067dc720f07a112ea075407b6ed9de965540504"} Sep 10 02:50:51.559759 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.559733 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:51.598610 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.598575 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfpct\" (UniqueName: \"kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct\") pod \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " Sep 10 02:50:51.598803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.598630 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle\") pod \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " Sep 10 02:50:51.598803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.598654 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util\") pod \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\" (UID: \"ce91915c-aec9-4a18-93ca-e9b590ac9ffe\") " Sep 10 02:50:51.599310 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.599284 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle" (OuterVolumeSpecName: "bundle") pod "ce91915c-aec9-4a18-93ca-e9b590ac9ffe" (UID: "ce91915c-aec9-4a18-93ca-e9b590ac9ffe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:50:51.600754 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.600730 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct" (OuterVolumeSpecName: "kube-api-access-kfpct") pod "ce91915c-aec9-4a18-93ca-e9b590ac9ffe" (UID: "ce91915c-aec9-4a18-93ca-e9b590ac9ffe"). InnerVolumeSpecName "kube-api-access-kfpct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:50:51.602451 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.602431 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util" (OuterVolumeSpecName: "util") pod "ce91915c-aec9-4a18-93ca-e9b590ac9ffe" (UID: "ce91915c-aec9-4a18-93ca-e9b590ac9ffe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:50:51.699425 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.699340 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfpct\" (UniqueName: \"kubernetes.io/projected/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-kube-api-access-kfpct\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:51.699425 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.699373 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-bundle\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:51.699425 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:51.699382 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce91915c-aec9-4a18-93ca-e9b590ac9ffe-util\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:50:52.440076 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:52.440045 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" Sep 10 02:50:52.440226 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:52.440042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540blv2h" event={"ID":"ce91915c-aec9-4a18-93ca-e9b590ac9ffe","Type":"ContainerDied","Data":"0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc"} Sep 10 02:50:52.440226 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:52.440168 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1dd531b2e51098454311f250a6cb7e713ff3c5aac3fd1bf441ea213ee77dcc" Sep 10 02:50:56.394043 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394003 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb"] Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394330 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="pull" Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394342 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="pull" Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394352 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="extract" Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394359 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="extract" Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394411 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="util" Sep 10 02:50:56.394444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394416 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="util" Sep 10 02:50:56.394627 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.394469 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce91915c-aec9-4a18-93ca-e9b590ac9ffe" containerName="extract" Sep 10 02:50:56.450831 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.450788 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb"] Sep 10 02:50:56.451011 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.450915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.454280 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.454254 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:50:56.455807 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.455785 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 02:50:56.463809 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.463784 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c8qj7\"" Sep 10 02:50:56.535404 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.535369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhxw\" (UniqueName: \"kubernetes.io/projected/8a97d664-685f-43ea-b38f-c6c122b7ae01-kube-api-access-mdhxw\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.535580 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.535430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97d664-685f-43ea-b38f-c6c122b7ae01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.636604 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.636564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97d664-685f-43ea-b38f-c6c122b7ae01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.636779 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.636629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhxw\" (UniqueName: \"kubernetes.io/projected/8a97d664-685f-43ea-b38f-c6c122b7ae01-kube-api-access-mdhxw\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.637006 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.636983 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97d664-685f-43ea-b38f-c6c122b7ae01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.647916 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.647846 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhxw\" (UniqueName: \"kubernetes.io/projected/8a97d664-685f-43ea-b38f-c6c122b7ae01-kube-api-access-mdhxw\") pod \"cert-manager-operator-controller-manager-55c444bf86-c8vvb\" (UID: \"8a97d664-685f-43ea-b38f-c6c122b7ae01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:56.761304 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:56.761268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" Sep 10 02:50:57.060231 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:57.060200 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb"] Sep 10 02:50:57.063835 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:50:57.063808 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a97d664_685f_43ea_b38f_c6c122b7ae01.slice/crio-b9966ce22d9d080fbd4dfb083ca79135d503b19906d9ca9397931004b8b9edbe WatchSource:0}: Error finding container b9966ce22d9d080fbd4dfb083ca79135d503b19906d9ca9397931004b8b9edbe: Status 404 returned error can't find the container with id b9966ce22d9d080fbd4dfb083ca79135d503b19906d9ca9397931004b8b9edbe Sep 10 02:50:57.464572 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:57.464471 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" event={"ID":"8a97d664-685f-43ea-b38f-c6c122b7ae01","Type":"ContainerStarted","Data":"b9966ce22d9d080fbd4dfb083ca79135d503b19906d9ca9397931004b8b9edbe"} Sep 10 02:50:59.477648 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.477551 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" event={"ID":"8a97d664-685f-43ea-b38f-c6c122b7ae01","Type":"ContainerStarted","Data":"a0a46d90eb23236388c317d437755e6136ea8462644dcc2bad9dc98108088851"} Sep 10 02:50:59.499296 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.499235 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c8vvb" podStartSLOduration=1.5357510620000001 podStartE2EDuration="3.499218254s" totalImagesPullingTime="1.963467192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:50:59.498433063 +0000 UTC m=+263.022289752" watchObservedRunningTime="2026-09-10 02:50:59.499218254 +0000 UTC m=+263.023074941" Sep 10 02:50:59.914748 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.914707 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bdhxf"] Sep 10 02:50:59.929696 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.929664 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:50:59.932395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.932375 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 02:50:59.941500 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.941475 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 02:50:59.943675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.943649 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bdhxf"] Sep 10 02:50:59.971417 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.971379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5p2\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-kube-api-access-vs5p2\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:50:59.971584 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:50:59.971467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.072230 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.072188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.072384 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.072255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5p2\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-kube-api-access-vs5p2\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.105196 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.105162 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.108913 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.108878 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5p2\" (UniqueName: \"kubernetes.io/projected/a7384aec-5254-400b-8a33-d8c9a4ec9ccd-kube-api-access-vs5p2\") pod \"cert-manager-webhook-6486ffd997-bdhxf\" (UID: \"a7384aec-5254-400b-8a33-d8c9a4ec9ccd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.250618 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.250584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:00.555958 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:00.555873 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bdhxf"] Sep 10 02:51:00.558954 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:51:00.558910 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7384aec_5254_400b_8a33_d8c9a4ec9ccd.slice/crio-93b73003c25cf1fd46a4a16ebfb5c982b9a7d476578e39fca4424a7cfae145ec WatchSource:0}: Error finding container 93b73003c25cf1fd46a4a16ebfb5c982b9a7d476578e39fca4424a7cfae145ec: Status 404 returned error can't find the container with id 93b73003c25cf1fd46a4a16ebfb5c982b9a7d476578e39fca4424a7cfae145ec Sep 10 02:51:01.489655 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:01.489619 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" event={"ID":"a7384aec-5254-400b-8a33-d8c9a4ec9ccd","Type":"ContainerStarted","Data":"93b73003c25cf1fd46a4a16ebfb5c982b9a7d476578e39fca4424a7cfae145ec"} Sep 10 02:51:03.500596 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:03.500559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" event={"ID":"a7384aec-5254-400b-8a33-d8c9a4ec9ccd","Type":"ContainerStarted","Data":"40a5dd3e5ee700aaa6f0f401a0270f46cd08b981738011edfa921d02638f8b26"} Sep 10 02:51:03.501035 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:03.500621 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:03.523441 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:03.523394 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" podStartSLOduration=1.806073595 podStartE2EDuration="4.523377371s" totalImagesPullingTime="2.717303776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:03.521876984 +0000 UTC m=+267.045733662" watchObservedRunningTime="2026-09-10 02:51:03.523377371 +0000 UTC m=+267.047234059" Sep 10 02:51:09.507729 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:09.507693 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bdhxf" Sep 10 02:51:12.658870 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.658831 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8t4dl"] Sep 10 02:51:12.662235 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.662213 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.664352 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.664333 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nw69k\"" Sep 10 02:51:12.677773 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.677746 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8t4dl"] Sep 10 02:51:12.782226 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.782190 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-bound-sa-token\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.782395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.782292 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qnn\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-kube-api-access-l2qnn\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.882995 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.882961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qnn\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-kube-api-access-l2qnn\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.883158 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.883023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-bound-sa-token\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.891799 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.891774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-bound-sa-token\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.891962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.891936 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qnn\" (UniqueName: \"kubernetes.io/projected/598f370f-b739-412f-842b-85eab96ca0b2-kube-api-access-l2qnn\") pod \"cert-manager-55b4d954d5-8t4dl\" (UID: \"598f370f-b739-412f-842b-85eab96ca0b2\") " pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:12.972190 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:12.972091 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8t4dl" Sep 10 02:51:13.252721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:13.252684 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8t4dl"] Sep 10 02:51:13.256007 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:51:13.255971 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598f370f_b739_412f_842b_85eab96ca0b2.slice/crio-416aa35375ccc2310486db9cd5553d6ccf68748f2da6a79ae4089fd7e239352f WatchSource:0}: Error finding container 416aa35375ccc2310486db9cd5553d6ccf68748f2da6a79ae4089fd7e239352f: Status 404 returned error can't find the container with id 416aa35375ccc2310486db9cd5553d6ccf68748f2da6a79ae4089fd7e239352f Sep 10 02:51:13.548018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:13.547920 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8t4dl" event={"ID":"598f370f-b739-412f-842b-85eab96ca0b2","Type":"ContainerStarted","Data":"74896b255fbce617d6aadecb978577a69306795d4fc085a9d5af89da7adbef4a"} Sep 10 02:51:13.548018 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:13.547966 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8t4dl" event={"ID":"598f370f-b739-412f-842b-85eab96ca0b2","Type":"ContainerStarted","Data":"416aa35375ccc2310486db9cd5553d6ccf68748f2da6a79ae4089fd7e239352f"} Sep 10 02:51:13.572038 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:13.571985 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8t4dl" podStartSLOduration=1.5719678670000001 podStartE2EDuration="1.571967867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:13.571483523 +0000 UTC m=+277.095340211" watchObservedRunningTime="2026-09-10 02:51:13.571967867 +0000 UTC m=+277.095824553" Sep 10 02:51:34.111568 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.111523 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w"] Sep 10 02:51:34.115776 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.115751 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.119156 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.119133 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 02:51:34.119288 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.119176 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 02:51:34.120042 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.120019 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 02:51:34.120178 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.120078 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:51:34.120178 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.120029 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gkvsz\"" Sep 10 02:51:34.120178 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.120119 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 02:51:34.127636 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.127611 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w"] Sep 10 02:51:34.166080 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.166039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-manager-config\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.166080 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.166088 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.166343 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.166194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.166343 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.166250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tsmh\" (UniqueName: \"kubernetes.io/projected/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-kube-api-access-7tsmh\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.267433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.267394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tsmh\" (UniqueName: \"kubernetes.io/projected/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-kube-api-access-7tsmh\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.267433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.267437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-manager-config\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.267745 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.267468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.267745 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.267528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.268195 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.268171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-manager-config\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.270196 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.270173 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.270283 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.270199 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.276222 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.276199 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tsmh\" (UniqueName: \"kubernetes.io/projected/c102fa5d-ba57-4644-bcde-a9e36c59a3bb-kube-api-access-7tsmh\") pod \"lws-controller-manager-6cd97c998c-q7b6w\" (UID: \"c102fa5d-ba57-4644-bcde-a9e36c59a3bb\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.425373 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.425290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:34.695761 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:34.695730 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w"] Sep 10 02:51:34.698107 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:51:34.698072 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc102fa5d_ba57_4644_bcde_a9e36c59a3bb.slice/crio-488d2bf7ab4c99cfb35de3679fa0849d6dfc71120a1327184b4a84aadc8a13d6 WatchSource:0}: Error finding container 488d2bf7ab4c99cfb35de3679fa0849d6dfc71120a1327184b4a84aadc8a13d6: Status 404 returned error can't find the container with id 488d2bf7ab4c99cfb35de3679fa0849d6dfc71120a1327184b4a84aadc8a13d6 Sep 10 02:51:35.652518 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:35.652471 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" event={"ID":"c102fa5d-ba57-4644-bcde-a9e36c59a3bb","Type":"ContainerStarted","Data":"488d2bf7ab4c99cfb35de3679fa0849d6dfc71120a1327184b4a84aadc8a13d6"} Sep 10 02:51:36.910134 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:36.910103 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:51:36.913507 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:36.913483 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:51:36.925246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:36.925219 2583 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 02:51:37.664838 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:37.664799 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" event={"ID":"c102fa5d-ba57-4644-bcde-a9e36c59a3bb","Type":"ContainerStarted","Data":"8729d58aaebd27b9ce75ab94a399666b92f15f7231e8a421545833586a17aeb5"} Sep 10 02:51:37.667034 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:37.665014 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:37.682284 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:37.682235 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" podStartSLOduration=1.597409135 podStartE2EDuration="3.682220649s" totalImagesPullingTime="2.084811514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:37.68023271 +0000 UTC m=+301.204089398" watchObservedRunningTime="2026-09-10 02:51:37.682220649 +0000 UTC m=+301.206077336" Sep 10 02:51:47.052029 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.051995 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2"] Sep 10 02:51:47.055693 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.055671 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.057721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.057700 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 02:51:47.057837 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.057818 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 02:51:47.058482 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.058454 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 02:51:47.058983 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.058965 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 02:51:47.059125 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.059047 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fcm22\"" Sep 10 02:51:47.093185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.093145 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2"] Sep 10 02:51:47.181121 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.181066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.181301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.181154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.181301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.181183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.181301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.181225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmlk\" (UniqueName: \"kubernetes.io/projected/1d31c9d4-984c-4738-b117-1db458959ccd-kube-api-access-vxmlk\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.181301 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.181291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.282458 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.282411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.282595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.282470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmlk\" (UniqueName: \"kubernetes.io/projected/1d31c9d4-984c-4738-b117-1db458959ccd-kube-api-access-vxmlk\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.282595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.282503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.282595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.282537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.282595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.282586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.283126 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.283089 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.284960 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.284939 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d31c9d4-984c-4738-b117-1db458959ccd-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.285081 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.285062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.285143 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.285089 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d31c9d4-984c-4738-b117-1db458959ccd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.291123 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.291102 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmlk\" (UniqueName: \"kubernetes.io/projected/1d31c9d4-984c-4738-b117-1db458959ccd-kube-api-access-vxmlk\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-cthp2\" (UID: \"1d31c9d4-984c-4738-b117-1db458959ccd\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.365407 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.365307 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:47.654797 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.654768 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2"] Sep 10 02:51:47.657522 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:51:47.657494 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d31c9d4_984c_4738_b117_1db458959ccd.slice/crio-2ec55ff9ce5883605259154f6906dcfa8113d7ca342ee0284e88967b0f4cde6b WatchSource:0}: Error finding container 2ec55ff9ce5883605259154f6906dcfa8113d7ca342ee0284e88967b0f4cde6b: Status 404 returned error can't find the container with id 2ec55ff9ce5883605259154f6906dcfa8113d7ca342ee0284e88967b0f4cde6b Sep 10 02:51:47.659533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.659514 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:51:47.714774 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:47.714744 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" event={"ID":"1d31c9d4-984c-4738-b117-1db458959ccd","Type":"ContainerStarted","Data":"2ec55ff9ce5883605259154f6906dcfa8113d7ca342ee0284e88967b0f4cde6b"} Sep 10 02:51:48.671843 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:48.671805 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-q7b6w" Sep 10 02:51:50.733348 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:50.733316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" event={"ID":"1d31c9d4-984c-4738-b117-1db458959ccd","Type":"ContainerStarted","Data":"0d8d29a6b46640460561ad5fa57c108029bb675a6891eba7c2ed8235779a3d26"} Sep 10 02:51:52.743498 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:52.743461 2583 generic.go:356] "Generic (PLEG): container finished" podID="1d31c9d4-984c-4738-b117-1db458959ccd" containerID="0d8d29a6b46640460561ad5fa57c108029bb675a6891eba7c2ed8235779a3d26" exitCode=0 Sep 10 02:51:52.743883 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:52.743538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" event={"ID":"1d31c9d4-984c-4738-b117-1db458959ccd","Type":"ContainerDied","Data":"0d8d29a6b46640460561ad5fa57c108029bb675a6891eba7c2ed8235779a3d26"} Sep 10 02:51:53.749655 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:53.749621 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" event={"ID":"1d31c9d4-984c-4738-b117-1db458959ccd","Type":"ContainerStarted","Data":"675b6cef4c8e14407e61763f8560497e3b7a8e1613b78ce54a4398ed24f8db80"} Sep 10 02:51:53.750060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:53.749842 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:51:53.775007 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:51:53.772975 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" podStartSLOduration=2.315826808 podStartE2EDuration="6.772956024s" totalImagesPullingTime="3.391134503s" totalInitContainerRuntime="1.065994713s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:53.76867612 +0000 UTC m=+317.292532808" watchObservedRunningTime="2026-09-10 02:51:53.772956024 +0000 UTC m=+317.296812711" Sep 10 02:52:04.757091 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:04.757057 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-cthp2" Sep 10 02:52:12.821516 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:12.821476 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf"] Sep 10 02:52:12.830995 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:12.830965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:12.833519 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:12.833490 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jgxl7\"" Sep 10 02:52:12.834156 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:12.833880 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 02:52:12.835112 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:12.835091 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf"] Sep 10 02:52:13.003737 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.003924 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.003924 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.003924 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrdm\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-kube-api-access-2rrdm\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.003924 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.004064 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.004064 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003967 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.004064 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.003992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.004064 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.004009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105333 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105506 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105506 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrdm\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-kube-api-access-2rrdm\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105506 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105987 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105987 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105987 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.105987 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.105819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.106185 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.106009 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.106391 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.106372 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.106491 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.106475 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.107999 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.107976 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.108403 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.108381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.120052 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.120025 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.123486 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.123464 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrdm\" (UniqueName: \"kubernetes.io/projected/f06d4ba4-1b0c-47f8-8dc2-08518fec0431-kube-api-access-2rrdm\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf\" (UID: \"f06d4ba4-1b0c-47f8-8dc2-08518fec0431\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.142276 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.142244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:13.421511 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.421433 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf"] Sep 10 02:52:13.424834 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:52:13.424806 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06d4ba4_1b0c_47f8_8dc2_08518fec0431.slice/crio-e36b7854f8ebff1ffb2ca92bae7922edaf78a3cba8348b206e00ce42aa8013e4 WatchSource:0}: Error finding container e36b7854f8ebff1ffb2ca92bae7922edaf78a3cba8348b206e00ce42aa8013e4: Status 404 returned error can't find the container with id e36b7854f8ebff1ffb2ca92bae7922edaf78a3cba8348b206e00ce42aa8013e4 Sep 10 02:52:13.842939 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:13.842905 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" event={"ID":"f06d4ba4-1b0c-47f8-8dc2-08518fec0431","Type":"ContainerStarted","Data":"e36b7854f8ebff1ffb2ca92bae7922edaf78a3cba8348b206e00ce42aa8013e4"} Sep 10 02:52:15.967243 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:15.967201 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:52:15.967511 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:15.967288 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:52:15.967511 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:15.967318 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:52:16.860898 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:16.860840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" event={"ID":"f06d4ba4-1b0c-47f8-8dc2-08518fec0431","Type":"ContainerStarted","Data":"f340144a07e04b3727ace9cf1481b08fce64bf08206a0b02fcd4154c42282b3b"} Sep 10 02:52:16.883142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:16.883091 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" podStartSLOduration=2.343037543 podStartE2EDuration="4.883072887s" totalImagesPullingTime="2.540035344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:16.881195024 +0000 UTC m=+340.405051746" watchObservedRunningTime="2026-09-10 02:52:16.883072887 +0000 UTC m=+340.406929632" Sep 10 02:52:17.142991 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:17.142909 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:17.148020 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:17.147995 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:17.865752 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:17.865720 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:17.872915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:17.872524 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf" Sep 10 02:52:24.576425 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.576395 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh"] Sep 10 02:52:24.580027 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.580005 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.582293 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.582271 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 02:52:24.582468 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.582292 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 02:52:24.582468 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.582275 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 02:52:24.582468 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.582427 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tg48t\"" Sep 10 02:52:24.593376 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.593352 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh"] Sep 10 02:52:24.604524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed26a76-df1a-4017-97ef-a59cd4ccd31a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604778 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmr9\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-kube-api-access-lgmr9\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604778 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604749 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604910 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604785 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.604910 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.604822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705244 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed26a76-df1a-4017-97ef-a59cd4ccd31a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705346 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmr9\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-kube-api-access-lgmr9\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705373 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.705643 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.705425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.706199 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.706170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.707778 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.707748 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed26a76-df1a-4017-97ef-a59cd4ccd31a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.707902 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.707836 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.707902 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.707845 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.707902 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.707889 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.717958 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.713828 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmr9\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-kube-api-access-lgmr9\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.717958 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.713901 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed26a76-df1a-4017-97ef-a59cd4ccd31a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sw9nh\" (UID: \"fed26a76-df1a-4017-97ef-a59cd4ccd31a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:24.890091 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:24.890009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:25.180671 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:25.180449 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh"] Sep 10 02:52:25.183426 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:52:25.183399 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed26a76_df1a_4017_97ef_a59cd4ccd31a.slice/crio-4f1416a3067cc3100e7c1d01e6080100599f397e6cb1db83d445476ada7c4d2a WatchSource:0}: Error finding container 4f1416a3067cc3100e7c1d01e6080100599f397e6cb1db83d445476ada7c4d2a: Status 404 returned error can't find the container with id 4f1416a3067cc3100e7c1d01e6080100599f397e6cb1db83d445476ada7c4d2a Sep 10 02:52:25.906310 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:25.906269 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" event={"ID":"fed26a76-df1a-4017-97ef-a59cd4ccd31a","Type":"ContainerStarted","Data":"4f1416a3067cc3100e7c1d01e6080100599f397e6cb1db83d445476ada7c4d2a"} Sep 10 02:52:27.642728 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:27.642688 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:52:27.643062 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:27.642764 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:52:27.917569 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:27.917474 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" event={"ID":"fed26a76-df1a-4017-97ef-a59cd4ccd31a","Type":"ContainerStarted","Data":"83d19fd2be09a12dfa1e3c072e7e6534f11af93e0c0b31612b0dd3cb881ef7be"} Sep 10 02:52:27.917569 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:27.917544 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:52:27.943317 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:27.943243 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" podStartSLOduration=1.486120581 podStartE2EDuration="3.943222182s" totalImagesPullingTime="2.457101601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:27.941994058 +0000 UTC m=+351.465850760" watchObservedRunningTime="2026-09-10 02:52:27.943222182 +0000 UTC m=+351.467078872" Sep 10 02:52:28.924595 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:52:28.924562 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sw9nh" Sep 10 02:53:26.089218 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.089182 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84ccbdcc64-w4cgx"] Sep 10 02:53:26.096630 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.096606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.115462 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.114896 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ccbdcc64-w4cgx"] Sep 10 02:53:26.142042 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-console-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142042 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-oauth-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-service-ca\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142181 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gth\" (UniqueName: \"kubernetes.io/projected/652f8d84-c710-4ff6-9e64-acd63a03da75-kube-api-access-72gth\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142206 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-trusted-ca-bundle\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.142374 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.142296 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-oauth-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243134 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-console-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243134 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-oauth-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-service-ca\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243212 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gth\" (UniqueName: \"kubernetes.io/projected/652f8d84-c710-4ff6-9e64-acd63a03da75-kube-api-access-72gth\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-trusted-ca-bundle\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.243393 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-oauth-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.244014 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.243988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-console-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.244122 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.244019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-service-ca\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.244122 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.244097 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-oauth-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.244122 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.244105 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652f8d84-c710-4ff6-9e64-acd63a03da75-trusted-ca-bundle\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.245652 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.245626 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-oauth-config\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.245755 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.245737 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/652f8d84-c710-4ff6-9e64-acd63a03da75-console-serving-cert\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.251667 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.251643 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gth\" (UniqueName: \"kubernetes.io/projected/652f8d84-c710-4ff6-9e64-acd63a03da75-kube-api-access-72gth\") pod \"console-84ccbdcc64-w4cgx\" (UID: \"652f8d84-c710-4ff6-9e64-acd63a03da75\") " pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.406288 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.406196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:26.680950 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:26.680923 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ccbdcc64-w4cgx"] Sep 10 02:53:26.683827 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:53:26.683798 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652f8d84_c710_4ff6_9e64_acd63a03da75.slice/crio-2a10c18b8dbc9057d5e86b62c32f8d99a44929170abb7b9f813ced53b1d201b4 WatchSource:0}: Error finding container 2a10c18b8dbc9057d5e86b62c32f8d99a44929170abb7b9f813ced53b1d201b4: Status 404 returned error can't find the container with id 2a10c18b8dbc9057d5e86b62c32f8d99a44929170abb7b9f813ced53b1d201b4 Sep 10 02:53:27.193683 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:27.193646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ccbdcc64-w4cgx" event={"ID":"652f8d84-c710-4ff6-9e64-acd63a03da75","Type":"ContainerStarted","Data":"bdf0e326ba2e05a9547f97e71d74b6bd552751a2174ca1a37c88d20c31035c8c"} Sep 10 02:53:27.194098 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:27.193695 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ccbdcc64-w4cgx" event={"ID":"652f8d84-c710-4ff6-9e64-acd63a03da75","Type":"ContainerStarted","Data":"2a10c18b8dbc9057d5e86b62c32f8d99a44929170abb7b9f813ced53b1d201b4"} Sep 10 02:53:27.216281 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:27.216222 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84ccbdcc64-w4cgx" podStartSLOduration=1.21620559 podStartE2EDuration="1.21620559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:53:27.21497032 +0000 UTC m=+410.738827009" watchObservedRunningTime="2026-09-10 02:53:27.21620559 +0000 UTC m=+410.740062276" Sep 10 02:53:36.406678 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:36.406637 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:36.407084 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:36.406711 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:36.411573 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:36.411545 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:37.242543 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:37.242515 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84ccbdcc64-w4cgx" Sep 10 02:53:37.299580 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:53:37.299539 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:54:02.321723 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.321623 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b5fb5ccb5-z2hll" podUID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" containerName="console" containerID="cri-o://bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075" gracePeriod=15 Sep 10 02:54:02.567423 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.567397 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:54:02.674505 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674408 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674505 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674452 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674505 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674509 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wplpw\" (UniqueName: \"kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674795 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674538 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674795 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674566 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674795 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674661 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.674795 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674745 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle\") pod \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\" (UID: \"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0\") " Sep 10 02:54:02.675029 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.674987 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:54:02.675135 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.675117 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.675203 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.675146 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:54:02.675203 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.675164 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config" (OuterVolumeSpecName: "console-config") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:54:02.675335 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.675305 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:54:02.676929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.676898 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:54:02.677128 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.677097 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:54:02.677128 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.677123 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw" (OuterVolumeSpecName: "kube-api-access-wplpw") pod "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" (UID: "4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0"). InnerVolumeSpecName "kube-api-access-wplpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:54:02.775979 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.775937 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-trusted-ca-bundle\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.775979 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.775969 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-oauth-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.775979 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.775978 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplpw\" (UniqueName: \"kubernetes.io/projected/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-kube-api-access-wplpw\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.775979 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.775987 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-serving-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.776242 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.775995 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:02.776242 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:02.776003 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0-console-oauth-config\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:54:03.371822 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.371735 2583 generic.go:356] "Generic (PLEG): container finished" podID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" containerID="bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075" exitCode=0 Sep 10 02:54:03.371822 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.371776 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fb5ccb5-z2hll" Sep 10 02:54:03.372269 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.371818 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fb5ccb5-z2hll" event={"ID":"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0","Type":"ContainerDied","Data":"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075"} Sep 10 02:54:03.372269 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.371885 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fb5ccb5-z2hll" event={"ID":"4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0","Type":"ContainerDied","Data":"c90dfce05cd616140d2193a1c4b04c672fbf36aa94e90cc3ac3af0d38d29d169"} Sep 10 02:54:03.372269 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.371906 2583 scope.go:117] "RemoveContainer" containerID="bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075" Sep 10 02:54:03.380806 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.380786 2583 scope.go:117] "RemoveContainer" containerID="bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075" Sep 10 02:54:03.381154 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:54:03.381133 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075\": container with ID starting with bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075 not found: ID does not exist" containerID="bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075" Sep 10 02:54:03.381213 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.381163 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075"} err="failed to get container status \"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075\": rpc error: code = NotFound desc = could not find container \"bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075\": container with ID starting with bad0930977666f693adba8671602ff66570567904422c91eb6f0aa709cda5075 not found: ID does not exist" Sep 10 02:54:03.389443 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.389419 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:54:03.392495 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:03.392467 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5fb5ccb5-z2hll"] Sep 10 02:54:04.999663 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:54:04.999627 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" path="/var/lib/kubelet/pods/4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0/volumes" Sep 10 02:55:22.765550 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.765515 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:22.766032 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.765835 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" containerName="console" Sep 10 02:55:22.766032 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.765846 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" containerName="console" Sep 10 02:55:22.766032 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.765934 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b20aaf1-eb66-4ac4-adc0-7c9a75e155a0" containerName="console" Sep 10 02:55:22.768988 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.768957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.773587 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.773562 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 02:55:22.773769 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.773744 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lg6lz\"" Sep 10 02:55:22.773891 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.773780 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 02:55:22.774352 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.774323 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 02:55:22.783570 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.783545 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:22.882623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.882577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.882813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.882696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.882813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.882725 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq7v\" (UniqueName: \"kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.958716 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.958677 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 02:55:22.962078 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.962061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:22.971470 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.971444 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 02:55:22.983885 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.983836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.984078 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.983944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.984078 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.983981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq7v\" (UniqueName: \"kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:22.984078 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:55:22.983996 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 02:55:22.984078 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:55:22.984068 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert podName:56862c3b-5f32-4451-975c-bbfe1f3003f4 nodeName:}" failed. No retries permitted until 2026-09-10 02:55:23.484045548 +0000 UTC m=+527.007902214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert") pod "maas-controller-b8c7cd48b-6t9bn" (UID: "56862c3b-5f32-4451-975c-bbfe1f3003f4") : secret "maas-controller-webhook-cert" not found Sep 10 02:55:22.984262 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:55:22.984089 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 10 02:55:22.984262 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:55:22.984157 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs podName:56862c3b-5f32-4451-975c-bbfe1f3003f4 nodeName:}" failed. No retries permitted until 2026-09-10 02:55:23.484142237 +0000 UTC m=+527.007998908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs") pod "maas-controller-b8c7cd48b-6t9bn" (UID: "56862c3b-5f32-4451-975c-bbfe1f3003f4") : secret "maas-controller-metrics-tls" not found Sep 10 02:55:22.992079 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:22.992054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq7v\" (UniqueName: \"kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.085022 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.084928 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5t98\" (UniqueName: \"kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.085191 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.085073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.085191 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.085120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.186507 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.186466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.186680 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.186518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.186680 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.186573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5t98\" (UniqueName: \"kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.189201 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.189169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.189337 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.189169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.195272 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.195245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5t98\" (UniqueName: \"kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98\") pod \"maas-controller-76d967fdff-gj4zj\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.272555 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.272511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:23.489577 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.489470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.489577 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.489575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.492054 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.492023 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.492158 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.492061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") pod \"maas-controller-b8c7cd48b-6t9bn\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.545142 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.545096 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 02:55:23.547511 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:55:23.547480 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc460ea6_63d7_48d5_b0c8_d85a7942e884.slice/crio-6b6bdc63c8aadfd650dcfb9e745157c1a1068631fd98f44760e381ee9d06836f WatchSource:0}: Error finding container 6b6bdc63c8aadfd650dcfb9e745157c1a1068631fd98f44760e381ee9d06836f: Status 404 returned error can't find the container with id 6b6bdc63c8aadfd650dcfb9e745157c1a1068631fd98f44760e381ee9d06836f Sep 10 02:55:23.678938 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.678900 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:23.746836 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.746759 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-gj4zj" event={"ID":"dc460ea6-63d7-48d5-b0c8-d85a7942e884","Type":"ContainerStarted","Data":"6b6bdc63c8aadfd650dcfb9e745157c1a1068631fd98f44760e381ee9d06836f"} Sep 10 02:55:23.950522 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:23.950495 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:23.952490 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:55:23.952463 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56862c3b_5f32_4451_975c_bbfe1f3003f4.slice/crio-ef89e2b9043f12606624c7be1515e32aebfb72c31fb43748610651d8981166f1 WatchSource:0}: Error finding container ef89e2b9043f12606624c7be1515e32aebfb72c31fb43748610651d8981166f1: Status 404 returned error can't find the container with id ef89e2b9043f12606624c7be1515e32aebfb72c31fb43748610651d8981166f1 Sep 10 02:55:24.757224 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:24.757182 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" event={"ID":"56862c3b-5f32-4451-975c-bbfe1f3003f4","Type":"ContainerStarted","Data":"ef89e2b9043f12606624c7be1515e32aebfb72c31fb43748610651d8981166f1"} Sep 10 02:55:26.772395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.772356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-gj4zj" event={"ID":"dc460ea6-63d7-48d5-b0c8-d85a7942e884","Type":"ContainerStarted","Data":"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4"} Sep 10 02:55:26.772902 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.772486 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:26.774194 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.774160 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" event={"ID":"56862c3b-5f32-4451-975c-bbfe1f3003f4","Type":"ContainerStarted","Data":"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78"} Sep 10 02:55:26.774342 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.774306 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:26.795199 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.795134 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76d967fdff-gj4zj" podStartSLOduration=2.177333584 podStartE2EDuration="4.795116135s" totalImagesPullingTime="2.617782551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:55:26.793962194 +0000 UTC m=+530.317818881" watchObservedRunningTime="2026-09-10 02:55:26.795116135 +0000 UTC m=+530.318972821" Sep 10 02:55:26.813825 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:26.813767 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" podStartSLOduration=2.5934478309999998 podStartE2EDuration="4.813742533s" totalImagesPullingTime="2.220294702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:55:26.811837692 +0000 UTC m=+530.335694382" watchObservedRunningTime="2026-09-10 02:55:26.813742533 +0000 UTC m=+530.337599220" Sep 10 02:55:37.781725 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:37.781598 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 02:55:37.782105 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:37.781796 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:37.846791 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:37.846760 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:37.847013 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:37.846968 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" podUID="56862c3b-5f32-4451-975c-bbfe1f3003f4" containerName="manager" containerID="cri-o://ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78" gracePeriod=30 Sep 10 02:55:38.099265 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.099239 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:38.224912 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.224850 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") pod \"56862c3b-5f32-4451-975c-bbfe1f3003f4\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " Sep 10 02:55:38.225107 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.225017 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gq7v\" (UniqueName: \"kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v\") pod \"56862c3b-5f32-4451-975c-bbfe1f3003f4\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " Sep 10 02:55:38.225107 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.225069 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") pod \"56862c3b-5f32-4451-975c-bbfe1f3003f4\" (UID: \"56862c3b-5f32-4451-975c-bbfe1f3003f4\") " Sep 10 02:55:38.227826 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.227791 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "56862c3b-5f32-4451-975c-bbfe1f3003f4" (UID: "56862c3b-5f32-4451-975c-bbfe1f3003f4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:55:38.227826 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.227810 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56862c3b-5f32-4451-975c-bbfe1f3003f4" (UID: "56862c3b-5f32-4451-975c-bbfe1f3003f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:55:38.228038 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.227832 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v" (OuterVolumeSpecName: "kube-api-access-6gq7v") pod "56862c3b-5f32-4451-975c-bbfe1f3003f4" (UID: "56862c3b-5f32-4451-975c-bbfe1f3003f4"). InnerVolumeSpecName "kube-api-access-6gq7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:55:38.326496 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.326459 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:55:38.326496 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.326491 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gq7v\" (UniqueName: \"kubernetes.io/projected/56862c3b-5f32-4451-975c-bbfe1f3003f4-kube-api-access-6gq7v\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:55:38.326496 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.326501 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56862c3b-5f32-4451-975c-bbfe1f3003f4-webhook-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:55:38.836789 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.836749 2583 generic.go:356] "Generic (PLEG): container finished" podID="56862c3b-5f32-4451-975c-bbfe1f3003f4" containerID="ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78" exitCode=0 Sep 10 02:55:38.837297 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.836796 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" Sep 10 02:55:38.837297 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.836832 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" event={"ID":"56862c3b-5f32-4451-975c-bbfe1f3003f4","Type":"ContainerDied","Data":"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78"} Sep 10 02:55:38.837297 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.836900 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8c7cd48b-6t9bn" event={"ID":"56862c3b-5f32-4451-975c-bbfe1f3003f4","Type":"ContainerDied","Data":"ef89e2b9043f12606624c7be1515e32aebfb72c31fb43748610651d8981166f1"} Sep 10 02:55:38.837297 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.836925 2583 scope.go:117] "RemoveContainer" containerID="ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78" Sep 10 02:55:38.845934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.845915 2583 scope.go:117] "RemoveContainer" containerID="ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78" Sep 10 02:55:38.846236 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:55:38.846217 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78\": container with ID starting with ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78 not found: ID does not exist" containerID="ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78" Sep 10 02:55:38.846310 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.846242 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78"} err="failed to get container status \"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78\": rpc error: code = NotFound desc = could not find container \"ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78\": container with ID starting with ad20c6c21ba467b9047ea1e42695b903fda1ecfb0d9b0091fcf1e15273e54c78 not found: ID does not exist" Sep 10 02:55:38.856990 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.856961 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:38.861245 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.861217 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b8c7cd48b-6t9bn"] Sep 10 02:55:38.999966 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:38.999925 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56862c3b-5f32-4451-975c-bbfe1f3003f4" path="/var/lib/kubelet/pods/56862c3b-5f32-4451-975c-bbfe1f3003f4/volumes" Sep 10 02:55:59.506432 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.506397 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 02:55:59.506975 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.506722 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56862c3b-5f32-4451-975c-bbfe1f3003f4" containerName="manager" Sep 10 02:55:59.506975 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.506733 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56862c3b-5f32-4451-975c-bbfe1f3003f4" containerName="manager" Sep 10 02:55:59.506975 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.506806 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="56862c3b-5f32-4451-975c-bbfe1f3003f4" containerName="manager" Sep 10 02:55:59.511244 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.511220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.513742 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.513716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq528\" (UniqueName: \"kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.513866 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.513801 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.513866 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.513835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.513967 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.513939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.514246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.514217 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 02:55:59.514246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.514235 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 02:55:59.514374 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.514320 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqv2s\"" Sep 10 02:55:59.514527 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.514503 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 02:55:59.514703 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.514657 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 02:55:59.518362 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.518339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 02:55:59.615053 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.615012 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq528\" (UniqueName: \"kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.615053 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.615058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.615305 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.615076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.615305 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.615109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.615798 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.615760 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.617721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.617699 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.617821 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.617724 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.623639 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.623614 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq528\" (UniqueName: \"kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528\") pod \"maas-api-77d77d65f8-5d8kk\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:55:59.823766 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:55:59.823681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:56:00.109577 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:00.109477 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 02:56:00.111715 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:56:00.111683 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c7e412_b3c3_443c_8a43_2ca4fc067c21.slice/crio-7e728925b938c4a215da4501aba2556fe1ac06a455810fb4366f6270c40e6f21 WatchSource:0}: Error finding container 7e728925b938c4a215da4501aba2556fe1ac06a455810fb4366f6270c40e6f21: Status 404 returned error can't find the container with id 7e728925b938c4a215da4501aba2556fe1ac06a455810fb4366f6270c40e6f21 Sep 10 02:56:00.950915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:00.950879 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" event={"ID":"93c7e412-b3c3-443c-8a43-2ca4fc067c21","Type":"ContainerStarted","Data":"7e728925b938c4a215da4501aba2556fe1ac06a455810fb4366f6270c40e6f21"} Sep 10 02:56:01.957480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:01.957387 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" event={"ID":"93c7e412-b3c3-443c-8a43-2ca4fc067c21","Type":"ContainerStarted","Data":"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b"} Sep 10 02:56:01.957832 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:01.957500 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:56:01.979246 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:01.979162 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" podStartSLOduration=1.437397228 podStartE2EDuration="2.979143505s" totalImagesPullingTime="1.541746277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:01.978817188 +0000 UTC m=+565.502673876" watchObservedRunningTime="2026-09-10 02:56:01.979143505 +0000 UTC m=+565.503000191" Sep 10 02:56:07.969515 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:07.969479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 02:56:30.543015 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.542976 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj"] Sep 10 02:56:30.546703 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.546683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.550841 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.550818 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 02:56:30.551033 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.550883 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dlt7k\"" Sep 10 02:56:30.551356 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.551340 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 02:56:30.551643 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.551628 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 02:56:30.573283 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.573254 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj"] Sep 10 02:56:30.693000 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.692956 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ffc10-7160-486b-a54b-38775171179d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.693000 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.693004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.693240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.693064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.693240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.693114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.693240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.693151 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.693240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.693234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqwv\" (UniqueName: \"kubernetes.io/projected/ae4ffc10-7160-486b-a54b-38775171179d-kube-api-access-9wqwv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794634 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ffc10-7160-486b-a54b-38775171179d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794634 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794809 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794731 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794809 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794915 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.794966 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.794937 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqwv\" (UniqueName: \"kubernetes.io/projected/ae4ffc10-7160-486b-a54b-38775171179d-kube-api-access-9wqwv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.795074 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.795055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.795144 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.795111 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.795144 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.795127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.797098 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.797072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4ffc10-7160-486b-a54b-38775171179d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.797211 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.797200 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4ffc10-7160-486b-a54b-38775171179d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.803194 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.803169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqwv\" (UniqueName: \"kubernetes.io/projected/ae4ffc10-7160-486b-a54b-38775171179d-kube-api-access-9wqwv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj\" (UID: \"ae4ffc10-7160-486b-a54b-38775171179d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:30.857381 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:30.857338 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:31.133631 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:31.133542 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj"] Sep 10 02:56:31.136700 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:56:31.136669 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4ffc10_7160_486b_a54b_38775171179d.slice/crio-cbcfaee7de0e0c9a11f10c0c173bae6107807c91f9bdbe33fa8653ccc1af4e25 WatchSource:0}: Error finding container cbcfaee7de0e0c9a11f10c0c173bae6107807c91f9bdbe33fa8653ccc1af4e25: Status 404 returned error can't find the container with id cbcfaee7de0e0c9a11f10c0c173bae6107807c91f9bdbe33fa8653ccc1af4e25 Sep 10 02:56:32.103573 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:32.103526 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" event={"ID":"ae4ffc10-7160-486b-a54b-38775171179d","Type":"ContainerStarted","Data":"cbcfaee7de0e0c9a11f10c0c173bae6107807c91f9bdbe33fa8653ccc1af4e25"} Sep 10 02:56:33.109930 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:33.109883 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" event={"ID":"ae4ffc10-7160-486b-a54b-38775171179d","Type":"ContainerStarted","Data":"4a92c9e714f4eef8abfd721048d87995e24903afd6cf232ce7469327aaa08223"} Sep 10 02:56:33.127960 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:33.127904 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" podStartSLOduration=2.027618236 podStartE2EDuration="3.127884999s" totalImagesPullingTime="1.100266763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:33.125987636 +0000 UTC m=+596.649844324" watchObservedRunningTime="2026-09-10 02:56:33.127884999 +0000 UTC m=+596.651741686" Sep 10 02:56:36.116683 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.116630 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx"] Sep 10 02:56:36.121556 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.121524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.123894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.123846 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 02:56:36.128587 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.128559 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx"] Sep 10 02:56:36.245673 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq97h\" (UniqueName: \"kubernetes.io/projected/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-kube-api-access-wq97h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.245673 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.245953 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.245953 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245759 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.245953 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.245953 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.245815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346624 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq97h\" (UniqueName: \"kubernetes.io/projected/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-kube-api-access-wq97h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346624 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346890 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346890 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346890 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.346890 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.346749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.347102 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.347077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.347140 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.347115 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.347191 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.347168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.349096 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.349075 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.349292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.349272 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.354189 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.354163 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq97h\" (UniqueName: \"kubernetes.io/projected/73f8ff10-5cbc-45d5-ab5c-db44d31f023b-kube-api-access-wq97h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m88kx\" (UID: \"73f8ff10-5cbc-45d5-ab5c-db44d31f023b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.434483 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.434396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:36.709339 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.709183 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx"] Sep 10 02:56:36.712009 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:56:36.711979 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f8ff10_5cbc_45d5_ab5c_db44d31f023b.slice/crio-c2eb646b278f506ae99b30a584e4d2f3ebd6a652625a6d6e81aff17919618657 WatchSource:0}: Error finding container c2eb646b278f506ae99b30a584e4d2f3ebd6a652625a6d6e81aff17919618657: Status 404 returned error can't find the container with id c2eb646b278f506ae99b30a584e4d2f3ebd6a652625a6d6e81aff17919618657 Sep 10 02:56:36.964592 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.964516 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:56:36.964592 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:36.964517 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 02:56:37.131137 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:37.131089 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" event={"ID":"73f8ff10-5cbc-45d5-ab5c-db44d31f023b","Type":"ContainerStarted","Data":"5940586fb53f708218703d1a3285fd0765611f0aa4e1ecce68ac63938b4453ea"} Sep 10 02:56:37.131505 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:37.131162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" event={"ID":"73f8ff10-5cbc-45d5-ab5c-db44d31f023b","Type":"ContainerStarted","Data":"c2eb646b278f506ae99b30a584e4d2f3ebd6a652625a6d6e81aff17919618657"} Sep 10 02:56:37.149112 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:37.149064 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" podStartSLOduration=0.963930491 podStartE2EDuration="1.149047284s" totalImagesPullingTime="185.116793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:37.148201751 +0000 UTC m=+600.672058438" watchObservedRunningTime="2026-09-10 02:56:37.149047284 +0000 UTC m=+600.672904013" Sep 10 02:56:40.858476 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:40.858435 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:40.860916 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:40.858517 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:40.871395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:40.871369 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:41.162892 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:41.162771 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj" Sep 10 02:56:46.434688 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:46.434653 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:46.435108 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:46.434722 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:46.447072 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:46.447041 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:56:47.197241 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:56:47.197214 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m88kx" Sep 10 02:57:09.808578 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.808541 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p"] Sep 10 02:57:09.813204 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.813180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.815297 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.815278 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 02:57:09.819837 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.819813 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p"] Sep 10 02:57:09.954233 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954192 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.954233 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.954433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954292 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.954433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.954433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:09.954533 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:09.954458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bc2\" (UniqueName: \"kubernetes.io/projected/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-kube-api-access-l4bc2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.055845 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.055806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.055907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bc2\" (UniqueName: \"kubernetes.io/projected/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-kube-api-access-l4bc2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.055967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.055997 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056060 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.056045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056325 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.056097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056409 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.056373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056537 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.056515 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.056600 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.056557 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.058076 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.058058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.058514 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.058485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.063300 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.063246 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bc2\" (UniqueName: \"kubernetes.io/projected/0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc-kube-api-access-l4bc2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cd92p\" (UID: \"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.125265 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.125228 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:10.398049 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.398022 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p"] Sep 10 02:57:10.400290 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:57:10.400256 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bebe3a2_6dde_4e70_b6c0_5c3db6eb5fbc.slice/crio-b5c81f4f1e66bd3db522b64160bd97f68e3a55c9005a146cdf01bce76c5d9b57 WatchSource:0}: Error finding container b5c81f4f1e66bd3db522b64160bd97f68e3a55c9005a146cdf01bce76c5d9b57: Status 404 returned error can't find the container with id b5c81f4f1e66bd3db522b64160bd97f68e3a55c9005a146cdf01bce76c5d9b57 Sep 10 02:57:10.401667 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:10.401648 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:57:11.317199 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:11.317159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" event={"ID":"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc","Type":"ContainerStarted","Data":"654bdb01d1dc61b3fc4c25b7b5ce392eb276e2407d32e15b9283012568f6da9e"} Sep 10 02:57:11.317564 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:11.317216 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" event={"ID":"0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc","Type":"ContainerStarted","Data":"b5c81f4f1e66bd3db522b64160bd97f68e3a55c9005a146cdf01bce76c5d9b57"} Sep 10 02:57:11.334756 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:11.334701 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" podStartSLOduration=2.156995063 podStartE2EDuration="2.334685204s" totalImagesPullingTime="177.690141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:57:11.332832804 +0000 UTC m=+634.856689491" watchObservedRunningTime="2026-09-10 02:57:11.334685204 +0000 UTC m=+634.858541891" Sep 10 02:57:20.126280 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:20.126226 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:20.126669 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:20.126308 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:20.139055 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:20.139029 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:57:20.372749 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:57:20.372719 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cd92p" Sep 10 02:58:53.821484 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.821397 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 02:58:53.825199 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.825166 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.827480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.827456 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b0f4da-openshift-default-dockercfg-9875n\"" Sep 10 02:58:53.838017 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.837988 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 02:58:53.936279 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936279 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936542 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7j5\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936542 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936391 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936542 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936542 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936542 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936806 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936535 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:53.936806 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:53.936568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038127 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038127 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7j5\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038379 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038571 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038617 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038663 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.038892 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.038698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.039244 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.039222 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.040636 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.040610 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.041011 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.040992 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.045709 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.045684 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.045804 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.045709 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7j5\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5\") pod \"e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.138004 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.137919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:54.418471 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.418438 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 02:58:54.420581 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:58:54.420552 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871779b9_64fe_47d5_a732_d0bea5edd793.slice/crio-72d7b8f94a18dbda7bb7590d9af37081e785a74336e98c346b98dc8472fb51dc WatchSource:0}: Error finding container 72d7b8f94a18dbda7bb7590d9af37081e785a74336e98c346b98dc8472fb51dc: Status 404 returned error can't find the container with id 72d7b8f94a18dbda7bb7590d9af37081e785a74336e98c346b98dc8472fb51dc Sep 10 02:58:54.422872 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.422820 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:58:54.422968 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.422931 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:58:54.423030 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.422985 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:58:54.835944 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.835909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerStarted","Data":"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14"} Sep 10 02:58:54.836390 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.835962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerStarted","Data":"72d7b8f94a18dbda7bb7590d9af37081e785a74336e98c346b98dc8472fb51dc"} Sep 10 02:58:54.854799 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:54.854741 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podStartSLOduration=1.854725997 podStartE2EDuration="1.854725997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:54.853067046 +0000 UTC m=+738.376923734" watchObservedRunningTime="2026-09-10 02:58:54.854725997 +0000 UTC m=+738.378582683" Sep 10 02:58:55.138940 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:55.138845 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:55.144152 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:55.144126 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:55.840999 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:55.840965 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:55.841905 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:55.841883 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:58:58.104234 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.104199 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 02:58:58.107683 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.107665 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.110137 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.110105 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-097e16-serving-cert\"" Sep 10 02:58:58.121260 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.121227 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 02:58:58.282029 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.281984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.282029 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.282033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvjm\" (UniqueName: \"kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.282240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.282116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.282240 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.282193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.383439 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.383344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.383594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.383442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.383594 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.383498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.383666 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:58:58.383600 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-serving-cert: secret "maas-api-e2e-worker-w0-097e16-serving-cert" not found Sep 10 02:58:58.383666 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.383625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvjm\" (UniqueName: \"kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.383734 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:58:58.383682 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls podName:60b5504d-a5ab-4648-b239-5ee95322b03d nodeName:}" failed. No retries permitted until 2026-09-10 02:58:58.883656829 +0000 UTC m=+742.407513526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls") pod "maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" (UID: "60b5504d-a5ab-4648-b239-5ee95322b03d") : secret "maas-api-e2e-worker-w0-097e16-serving-cert" not found Sep 10 02:58:58.384157 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.384137 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.386110 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.386085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.393689 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.393656 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvjm\" (UniqueName: \"kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.889433 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.889388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:58.892094 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:58.892064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:59.024282 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.024243 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:59.288200 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.288163 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 02:58:59.296535 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.294751 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.297548 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.297516 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e6e6b8-serving-cert\"" Sep 10 02:58:59.304455 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.304410 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 02:58:59.306586 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.306489 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 02:58:59.309135 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:58:59.309096 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b5504d_a5ab_4648_b239_5ee95322b03d.slice/crio-ebc0a12b29f593cb7b47f933d6aa655ba19afdf180fd1c8c3525c6bb2d2f9c4f WatchSource:0}: Error finding container ebc0a12b29f593cb7b47f933d6aa655ba19afdf180fd1c8c3525c6bb2d2f9c4f: Status 404 returned error can't find the container with id ebc0a12b29f593cb7b47f933d6aa655ba19afdf180fd1c8c3525c6bb2d2f9c4f Sep 10 02:58:59.409189 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.409141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.409521 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.409496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.409707 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.409687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.409938 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.409920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.510645 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.510609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.510811 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.510705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.510811 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.510741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.510811 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.510772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.510993 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:58:59.510774 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-serving-cert: secret "maas-api-e2e-worker-w4-e6e6b8-serving-cert" not found Sep 10 02:58:59.510993 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:58:59.510950 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls podName:89003daf-f265-429b-b174-c2d6cc0e5bd2 nodeName:}" failed. No retries permitted until 2026-09-10 02:59:00.010929478 +0000 UTC m=+743.534786150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls") pod "maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" (UID: "89003daf-f265-429b-b174-c2d6cc0e5bd2") : secret "maas-api-e2e-worker-w4-e6e6b8-serving-cert" not found Sep 10 02:58:59.511480 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.511457 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.513265 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.513244 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.519045 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.519021 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:58:59.864277 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.864243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" event={"ID":"60b5504d-a5ab-4648-b239-5ee95322b03d","Type":"ContainerStarted","Data":"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827"} Sep 10 02:58:59.864444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.864302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" event={"ID":"60b5504d-a5ab-4648-b239-5ee95322b03d","Type":"ContainerStarted","Data":"ebc0a12b29f593cb7b47f933d6aa655ba19afdf180fd1c8c3525c6bb2d2f9c4f"} Sep 10 02:58:59.864444 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.864343 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:58:59.889715 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:58:59.889659 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" podStartSLOduration=1.431712348 podStartE2EDuration="1.889641734s" totalImagesPullingTime="457.929386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:59.887013349 +0000 UTC m=+743.410870037" watchObservedRunningTime="2026-09-10 02:58:59.889641734 +0000 UTC m=+743.413498522" Sep 10 02:59:00.014239 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.014204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:59:00.016927 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.016897 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:59:00.213158 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.213054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:59:00.250055 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.249992 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 02:59:00.254547 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.254516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.257085 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.256768 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b0f4da-serving-cert\"" Sep 10 02:59:00.263412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.263188 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 02:59:00.316631 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.316582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r826f\" (UniqueName: \"kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.316631 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.316618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.317206 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.316694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.317206 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.316721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.417889 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.417822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.418089 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.417940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r826f\" (UniqueName: \"kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.418089 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.417959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.418089 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.418036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.418208 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:00.418174 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-serving-cert: secret "maas-api-e2e-inf-a-b0f4da-serving-cert" not found Sep 10 02:59:00.418246 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:00.418240 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls podName:6617bdd1-553e-4d50-95c3-693aafe448b3 nodeName:}" failed. No retries permitted until 2026-09-10 02:59:00.918224102 +0000 UTC m=+744.442080772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls") pod "maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" (UID: "6617bdd1-553e-4d50-95c3-693aafe448b3") : secret "maas-api-e2e-inf-a-b0f4da-serving-cert" not found Sep 10 02:59:00.418660 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.418642 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.420416 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.420388 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.426556 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.426520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r826f\" (UniqueName: \"kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.497395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.497365 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 02:59:00.499315 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:00.499287 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89003daf_f265_429b_b174_c2d6cc0e5bd2.slice/crio-dbfecd78bddb865f93310630dbf1b6cd9d80f2ca546caddbe7dd3feeaefeecf0 WatchSource:0}: Error finding container dbfecd78bddb865f93310630dbf1b6cd9d80f2ca546caddbe7dd3feeaefeecf0: Status 404 returned error can't find the container with id dbfecd78bddb865f93310630dbf1b6cd9d80f2ca546caddbe7dd3feeaefeecf0 Sep 10 02:59:00.872288 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.872251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" event={"ID":"89003daf-f265-429b-b174-c2d6cc0e5bd2","Type":"ContainerStarted","Data":"dbfecd78bddb865f93310630dbf1b6cd9d80f2ca546caddbe7dd3feeaefeecf0"} Sep 10 02:59:00.922250 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.922210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:00.924964 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:00.924943 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:01.171100 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.171019 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:01.448499 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.448461 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 02:59:01.449958 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:01.449933 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617bdd1_553e_4d50_95c3_693aafe448b3.slice/crio-08dce3f87623e18e14d28be6b5e3b094692d12dff2d1cb73ffdbf20a73200167 WatchSource:0}: Error finding container 08dce3f87623e18e14d28be6b5e3b094692d12dff2d1cb73ffdbf20a73200167: Status 404 returned error can't find the container with id 08dce3f87623e18e14d28be6b5e3b094692d12dff2d1cb73ffdbf20a73200167 Sep 10 02:59:01.879054 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.879021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" event={"ID":"89003daf-f265-429b-b174-c2d6cc0e5bd2","Type":"ContainerStarted","Data":"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0"} Sep 10 02:59:01.879216 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.879204 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:59:01.880503 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.880469 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" event={"ID":"6617bdd1-553e-4d50-95c3-693aafe448b3","Type":"ContainerStarted","Data":"08dce3f87623e18e14d28be6b5e3b094692d12dff2d1cb73ffdbf20a73200167"} Sep 10 02:59:01.897602 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:01.897553 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" podStartSLOduration=2.49313318 podStartE2EDuration="2.897535678s" totalImagesPullingTime="404.402498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.895562941 +0000 UTC m=+745.419419629" watchObservedRunningTime="2026-09-10 02:59:01.897535678 +0000 UTC m=+745.421392365" Sep 10 02:59:02.888045 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:02.888005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" event={"ID":"6617bdd1-553e-4d50-95c3-693aafe448b3","Type":"ContainerStarted","Data":"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61"} Sep 10 02:59:02.888458 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:02.888211 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:02.904721 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:02.904661 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" podStartSLOduration=2.5272921999999998 podStartE2EDuration="2.904643916s" totalImagesPullingTime="377.351716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:02.903156822 +0000 UTC m=+746.427013510" watchObservedRunningTime="2026-09-10 02:59:02.904643916 +0000 UTC m=+746.428500603" Sep 10 02:59:05.877342 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:05.877308 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 02:59:06.347495 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.347458 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:06.351618 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.351591 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.353705 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.353680 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-644ff039-serving-cert\"" Sep 10 02:59:06.361875 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.361837 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:06.371783 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.371751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5bb\" (UniqueName: \"kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.372357 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.372179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.372357 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.372218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.372357 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.372267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.473169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.473127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.473169 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.473170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.473438 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.473196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.473438 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.473326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5bb\" (UniqueName: \"kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.473962 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.473937 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.475982 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.475959 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.476037 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.475959 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.480674 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.480624 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5bb\" (UniqueName: \"kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb\") pod \"maas-api-e2e-ait-644ff039-5b5d995445-7mgrd\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.665675 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.665583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:06.943650 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:06.943621 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:06.946290 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:06.946261 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b12d5f_9828_48d3_9104_c27aeb723d3e.slice/crio-0e621e5001d4390f217ef4db31bdf0f11ec8caa79bd5125ae33c2c927dde0efa WatchSource:0}: Error finding container 0e621e5001d4390f217ef4db31bdf0f11ec8caa79bd5125ae33c2c927dde0efa: Status 404 returned error can't find the container with id 0e621e5001d4390f217ef4db31bdf0f11ec8caa79bd5125ae33c2c927dde0efa Sep 10 02:59:07.898483 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:07.898434 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 02:59:07.916614 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:07.916479 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" event={"ID":"63b12d5f-9828-48d3-9104-c27aeb723d3e","Type":"ContainerStarted","Data":"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf"} Sep 10 02:59:07.916614 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:07.916571 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" event={"ID":"63b12d5f-9828-48d3-9104-c27aeb723d3e","Type":"ContainerStarted","Data":"0e621e5001d4390f217ef4db31bdf0f11ec8caa79bd5125ae33c2c927dde0efa"} Sep 10 02:59:07.917004 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:07.916941 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:07.940757 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:07.940694 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" podStartSLOduration=1.582250682 podStartE2EDuration="1.94067335s" totalImagesPullingTime="358.422668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:07.937886513 +0000 UTC m=+751.461743203" watchObservedRunningTime="2026-09-10 02:59:07.94067335 +0000 UTC m=+751.464530080" Sep 10 02:59:08.179292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.179197 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 02:59:08.190890 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.187162 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.199920 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.199878 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 02:59:08.200223 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.200145 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-5ab28b-serving-cert\"" Sep 10 02:59:08.305483 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.305437 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.305769 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.305746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.306016 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.305985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.306093 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.306041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjbp\" (UniqueName: \"kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.407682 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.407630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.407911 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.407691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.407911 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.407748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.407911 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.407781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjbp\" (UniqueName: \"kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.408086 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:08.408063 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-serving-cert: secret "maas-api-e2e-autores-5ab28b-serving-cert" not found Sep 10 02:59:08.408143 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:08.408129 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls podName:8b3783d7-6800-417c-95ed-bc7ec1bdd0db nodeName:}" failed. No retries permitted until 2026-09-10 02:59:08.908111419 +0000 UTC m=+752.431968090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls") pod "maas-api-e2e-autores-5ab28b-796498876b-84qtl" (UID: "8b3783d7-6800-417c-95ed-bc7ec1bdd0db") : secret "maas-api-e2e-autores-5ab28b-serving-cert" not found Sep 10 02:59:08.408747 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.408713 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.410969 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.410937 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.416072 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.416044 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjbp\" (UniqueName: \"kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.912453 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.912421 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 02:59:08.915343 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.915320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:08.917957 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:08.917925 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") pod \"maas-api-e2e-autores-5ab28b-796498876b-84qtl\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:09.118260 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:09.118210 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:09.395384 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:09.395339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 02:59:09.403966 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:09.403842 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3783d7_6800_417c_95ed_bc7ec1bdd0db.slice/crio-3f7e959c700e93e51594fe9c09627fa06d5f51c8d3d720b8ba877c401b2d7127 WatchSource:0}: Error finding container 3f7e959c700e93e51594fe9c09627fa06d5f51c8d3d720b8ba877c401b2d7127: Status 404 returned error can't find the container with id 3f7e959c700e93e51594fe9c09627fa06d5f51c8d3d720b8ba877c401b2d7127 Sep 10 02:59:09.928292 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:09.928258 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" event={"ID":"8b3783d7-6800-417c-95ed-bc7ec1bdd0db","Type":"ContainerStarted","Data":"3f7e959c700e93e51594fe9c09627fa06d5f51c8d3d720b8ba877c401b2d7127"} Sep 10 02:59:10.935356 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:10.935312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" event={"ID":"8b3783d7-6800-417c-95ed-bc7ec1bdd0db","Type":"ContainerStarted","Data":"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8"} Sep 10 02:59:10.935831 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:10.935573 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:10.952788 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:10.952733 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" podStartSLOduration=2.452221577 podStartE2EDuration="2.952717677s" totalImagesPullingTime="500.4961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:10.951465413 +0000 UTC m=+754.475322097" watchObservedRunningTime="2026-09-10 02:59:10.952717677 +0000 UTC m=+754.476574364" Sep 10 02:59:13.928560 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:13.928528 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:16.947894 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.947802 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 02:59:16.972813 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.972776 2583 generic.go:356] "Generic (PLEG): container finished" podID="871779b9-64fe-47d5-a732-d0bea5edd793" containerID="072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14" exitCode=0 Sep 10 02:59:16.973017 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.972919 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerDied","Data":"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14"} Sep 10 02:59:16.973483 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.973448 2583 scope.go:117] "RemoveContainer" containerID="072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14" Sep 10 02:59:16.975144 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.975003 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:16.975144 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.975090 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:16.975477 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:16.975153 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:17.980923 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:17.980886 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerStarted","Data":"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764"} Sep 10 02:59:18.138940 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:18.138904 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:59:18.139213 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:18.139186 2583 patch_prober.go:28] interesting pod/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 02:59:18.139306 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:18.139246 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 02:59:19.138907 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:19.138842 2583 patch_prober.go:28] interesting pod/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 02:59:19.139306 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:19.138953 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 02:59:20.138718 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:20.138682 2583 patch_prober.go:28] interesting pod/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 02:59:20.138931 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:20.138778 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 02:59:21.138526 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:21.138492 2583 patch_prober.go:28] interesting pod/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 02:59:21.138705 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:21.138550 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 02:59:22.138801 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:22.138761 2583 patch_prober.go:28] interesting pod/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 10 02:59:22.139252 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:22.138824 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 10 02:59:23.242712 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:23.242673 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:59:23.243225 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:23.243200 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:59:23.244079 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:23.244058 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 02:59:32.237403 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.237358 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 02:59:32.241676 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.241649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.243929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.243904 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ce5f9d-serving-cert\"" Sep 10 02:59:32.251202 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.251172 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 02:59:32.357119 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.357077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.357438 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.357415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk854\" (UniqueName: \"kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.357536 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.357491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.357615 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.357537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.458273 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.458228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.458273 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.458279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.458537 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.458323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.458537 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.458370 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk854\" (UniqueName: \"kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.458537 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:32.458472 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-serving-cert: secret "maas-api-e2e-inf-b-ce5f9d-serving-cert" not found Sep 10 02:59:32.458785 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:32.458544 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls podName:0c8404c5-c81c-4854-b86b-90de30fd5fe1 nodeName:}" failed. No retries permitted until 2026-09-10 02:59:32.958523984 +0000 UTC m=+776.482380653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls") pod "maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" (UID: "0c8404c5-c81c-4854-b86b-90de30fd5fe1") : secret "maas-api-e2e-inf-b-ce5f9d-serving-cert" not found Sep 10 02:59:32.458927 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.458906 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.460795 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.460772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.466305 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.466280 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk854\" (UniqueName: \"kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.962337 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.962287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:32.965191 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:32.965146 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:33.155160 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:33.155114 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:33.429144 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:33.429108 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 02:59:33.433100 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:33.433074 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8404c5_c81c_4854_b86b_90de30fd5fe1.slice/crio-f06f3860f03eef0ff4a41d634746e57221d5647f863086f0f33bcfdc007c1d14 WatchSource:0}: Error finding container f06f3860f03eef0ff4a41d634746e57221d5647f863086f0f33bcfdc007c1d14: Status 404 returned error can't find the container with id f06f3860f03eef0ff4a41d634746e57221d5647f863086f0f33bcfdc007c1d14 Sep 10 02:59:34.085570 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:34.085532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" event={"ID":"0c8404c5-c81c-4854-b86b-90de30fd5fe1","Type":"ContainerStarted","Data":"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a"} Sep 10 02:59:34.085783 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:34.085592 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" event={"ID":"0c8404c5-c81c-4854-b86b-90de30fd5fe1","Type":"ContainerStarted","Data":"f06f3860f03eef0ff4a41d634746e57221d5647f863086f0f33bcfdc007c1d14"} Sep 10 02:59:34.085783 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:34.085652 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:34.102891 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:34.102821 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" podStartSLOduration=1.539700711 podStartE2EDuration="2.102803903s" totalImagesPullingTime="563.103192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:34.101494487 +0000 UTC m=+777.625351175" watchObservedRunningTime="2026-09-10 02:59:34.102803903 +0000 UTC m=+777.626660589" Sep 10 02:59:40.096395 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:40.096363 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 02:59:47.256755 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.256715 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:47.257330 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.256995 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" podUID="63b12d5f-9828-48d3-9104-c27aeb723d3e" containerName="maas-api" containerID="cri-o://c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf" gracePeriod=30 Sep 10 02:59:47.522651 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.522586 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:47.612455 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.612422 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs\") pod \"63b12d5f-9828-48d3-9104-c27aeb723d3e\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " Sep 10 02:59:47.612635 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.612517 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn5bb\" (UniqueName: \"kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb\") pod \"63b12d5f-9828-48d3-9104-c27aeb723d3e\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " Sep 10 02:59:47.612635 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.612552 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls\") pod \"63b12d5f-9828-48d3-9104-c27aeb723d3e\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " Sep 10 02:59:47.612635 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.612596 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca\") pod \"63b12d5f-9828-48d3-9104-c27aeb723d3e\" (UID: \"63b12d5f-9828-48d3-9104-c27aeb723d3e\") " Sep 10 02:59:47.613085 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.613051 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63b12d5f-9828-48d3-9104-c27aeb723d3e" (UID: "63b12d5f-9828-48d3-9104-c27aeb723d3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:59:47.614839 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.614806 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63b12d5f-9828-48d3-9104-c27aeb723d3e" (UID: "63b12d5f-9828-48d3-9104-c27aeb723d3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:59:47.614981 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.614851 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb" (OuterVolumeSpecName: "kube-api-access-pn5bb") pod "63b12d5f-9828-48d3-9104-c27aeb723d3e" (UID: "63b12d5f-9828-48d3-9104-c27aeb723d3e"). InnerVolumeSpecName "kube-api-access-pn5bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:59:47.615232 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.615208 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63b12d5f-9828-48d3-9104-c27aeb723d3e" (UID: "63b12d5f-9828-48d3-9104-c27aeb723d3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:59:47.713681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.713632 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b12d5f-9828-48d3-9104-c27aeb723d3e-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:59:47.713681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.713672 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:59:47.713681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.713682 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn5bb\" (UniqueName: \"kubernetes.io/projected/63b12d5f-9828-48d3-9104-c27aeb723d3e-kube-api-access-pn5bb\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:59:47.713681 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:47.713691 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b12d5f-9828-48d3-9104-c27aeb723d3e-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 02:59:48.160304 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.160269 2583 generic.go:356] "Generic (PLEG): container finished" podID="63b12d5f-9828-48d3-9104-c27aeb723d3e" containerID="c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf" exitCode=0 Sep 10 02:59:48.160513 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.160313 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" Sep 10 02:59:48.160513 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.160346 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" event={"ID":"63b12d5f-9828-48d3-9104-c27aeb723d3e","Type":"ContainerDied","Data":"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf"} Sep 10 02:59:48.160513 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.160406 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd" event={"ID":"63b12d5f-9828-48d3-9104-c27aeb723d3e","Type":"ContainerDied","Data":"0e621e5001d4390f217ef4db31bdf0f11ec8caa79bd5125ae33c2c927dde0efa"} Sep 10 02:59:48.160513 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.160435 2583 scope.go:117] "RemoveContainer" containerID="c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf" Sep 10 02:59:48.170397 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.170370 2583 scope.go:117] "RemoveContainer" containerID="c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf" Sep 10 02:59:48.170907 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:48.170885 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf\": container with ID starting with c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf not found: ID does not exist" containerID="c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf" Sep 10 02:59:48.170998 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.170914 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf"} err="failed to get container status \"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf\": rpc error: code = NotFound desc = could not find container \"c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf\": container with ID starting with c170dcf8a3a39da50cf660ee887c08762e62fea28bac90df98f724e229a91daf not found: ID does not exist" Sep 10 02:59:48.189329 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.189296 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:48.201046 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.201010 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-644ff039-5b5d995445-7mgrd"] Sep 10 02:59:48.755582 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.755532 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 02:59:48.756214 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.756192 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b12d5f-9828-48d3-9104-c27aeb723d3e" containerName="maas-api" Sep 10 02:59:48.756259 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.756219 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b12d5f-9828-48d3-9104-c27aeb723d3e" containerName="maas-api" Sep 10 02:59:48.756380 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.756367 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b12d5f-9828-48d3-9104-c27aeb723d3e" containerName="maas-api" Sep 10 02:59:48.761267 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.761244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.763407 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.763381 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-387ddcaa-gw-openshift-default-dockercfg-qlmrp\"" Sep 10 02:59:48.771705 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.771678 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 02:59:48.825978 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.825941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826173 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.825988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75flh\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826173 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826017 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826173 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826173 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826272 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.826399 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.826306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927321 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927290 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927524 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75flh\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927803 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927606 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.927994 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.927967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.928132 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.928105 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.928197 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.928137 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.928251 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.928191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.928351 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.928334 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.929880 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.929850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.930117 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.930101 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.934702 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.934676 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:48.934851 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:48.934829 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75flh\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh\") pod \"e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:49.000139 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.000105 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b12d5f-9828-48d3-9104-c27aeb723d3e" path="/var/lib/kubelet/pods/63b12d5f-9828-48d3-9104-c27aeb723d3e/volumes" Sep 10 02:59:49.073076 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.072981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:49.350776 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.350699 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 02:59:49.354403 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:49.354369 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c537f_8805_49d2_865b_1f2f5539f7f2.slice/crio-d729211a3a9d254e4ce29f521f4424c7d9fa0a4769a3ded10c93b166f61224d2 WatchSource:0}: Error finding container d729211a3a9d254e4ce29f521f4424c7d9fa0a4769a3ded10c93b166f61224d2: Status 404 returned error can't find the container with id d729211a3a9d254e4ce29f521f4424c7d9fa0a4769a3ded10c93b166f61224d2 Sep 10 02:59:49.356976 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.356933 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:49.357073 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.357028 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:49.357136 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:49.357080 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 02:59:50.176662 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:50.176617 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerStarted","Data":"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5"} Sep 10 02:59:50.177063 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:50.176673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerStarted","Data":"d729211a3a9d254e4ce29f521f4424c7d9fa0a4769a3ded10c93b166f61224d2"} Sep 10 02:59:50.198171 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:50.198118 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podStartSLOduration=2.198100663 podStartE2EDuration="2.198100663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:50.197489092 +0000 UTC m=+793.721345783" watchObservedRunningTime="2026-09-10 02:59:50.198100663 +0000 UTC m=+793.721957349" Sep 10 02:59:51.073194 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.073160 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:51.078509 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.078485 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:51.182623 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.182599 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:51.183584 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.183560 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 02:59:51.771982 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.771943 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 02:59:51.775988 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.775962 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.777933 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.777908 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-387ddcaa-serving-cert\"" Sep 10 02:59:51.786002 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.785554 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 02:59:51.856723 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.856684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mh2\" (UniqueName: \"kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.856929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.856799 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.856929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.856845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.856929 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.856881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.958411 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.958372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.958589 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.958419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.958589 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:51.958527 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-serving-cert: secret "maas-api-e2e-migrate-387ddcaa-serving-cert" not found Sep 10 02:59:51.958688 ip-10-0-134-225 kubenswrapper[2583]: E0910 02:59:51.958591 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls podName:94510e4d-5379-4215-8728-1778f47979cc nodeName:}" failed. No retries permitted until 2026-09-10 02:59:52.45857285 +0000 UTC m=+795.982429515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls") pod "maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" (UID: "94510e4d-5379-4215-8728-1778f47979cc") : secret "maas-api-e2e-migrate-387ddcaa-serving-cert" not found Sep 10 02:59:51.958688 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.958529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.958764 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.958718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mh2\" (UniqueName: \"kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.959242 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.959224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.960920 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.960903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:51.968957 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:51.968930 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mh2\" (UniqueName: \"kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:52.463934 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:52.463899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:52.466412 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:52.466388 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") pod \"maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:52.689688 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:52.689648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:52.967022 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:52.966988 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 02:59:52.969634 ip-10-0-134-225 kubenswrapper[2583]: W0910 02:59:52.969572 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94510e4d_5379_4215_8728_1778f47979cc.slice/crio-8b5d120abdc3a5dcab11d1d6b4e7170786033ec898f7d96376056f8a30a2b89b WatchSource:0}: Error finding container 8b5d120abdc3a5dcab11d1d6b4e7170786033ec898f7d96376056f8a30a2b89b: Status 404 returned error can't find the container with id 8b5d120abdc3a5dcab11d1d6b4e7170786033ec898f7d96376056f8a30a2b89b Sep 10 02:59:53.194845 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:53.194808 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" event={"ID":"94510e4d-5379-4215-8728-1778f47979cc","Type":"ContainerStarted","Data":"8b5d120abdc3a5dcab11d1d6b4e7170786033ec898f7d96376056f8a30a2b89b"} Sep 10 02:59:54.201396 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:54.201362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" event={"ID":"94510e4d-5379-4215-8728-1778f47979cc","Type":"ContainerStarted","Data":"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97"} Sep 10 02:59:54.201804 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:54.201431 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 02:59:54.221262 ip-10-0-134-225 kubenswrapper[2583]: I0910 02:59:54.221210 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" podStartSLOduration=2.765373431 podStartE2EDuration="3.221193484s" totalImagesPullingTime="455.820053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:54.219112772 +0000 UTC m=+797.742969460" watchObservedRunningTime="2026-09-10 02:59:54.221193484 +0000 UTC m=+797.745050171" Sep 10 03:00:00.212773 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:00.212691 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 03:00:13.322793 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.322755 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerID="58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5" exitCode=0 Sep 10 03:00:13.323348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.322831 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerDied","Data":"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5"} Sep 10 03:00:13.323444 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.323427 2583 scope.go:117] "RemoveContainer" containerID="58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5" Sep 10 03:00:13.324599 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.324569 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:00:13.324667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.324641 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:00:13.324704 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:13.324671 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:00:14.330309 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:14.330273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerStarted","Data":"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9"} Sep 10 03:00:15.074008 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:15.073975 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:15.074241 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:15.074219 2583 patch_prober.go:28] interesting pod/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 03:00:15.074302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:15.074282 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 03:00:16.074158 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:16.074119 2583 patch_prober.go:28] interesting pod/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 03:00:16.074547 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:16.074180 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 03:00:17.074234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:17.074192 2583 patch_prober.go:28] interesting pod/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 03:00:17.074639 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:17.074255 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 03:00:18.073739 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:18.073689 2583 patch_prober.go:28] interesting pod/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 03:00:18.073985 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:18.073780 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 03:00:19.073963 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:19.073921 2583 patch_prober.go:28] interesting pod/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 03:00:19.074364 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:19.073980 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 03:00:20.078216 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:20.078180 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:20.078594 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:20.078500 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:20.079133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:20.079114 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:23.056522 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.056478 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 03:00:23.056976 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.056746 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" podUID="94510e4d-5379-4215-8728-1778f47979cc" containerName="maas-api" containerID="cri-o://a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97" gracePeriod=30 Sep 10 03:00:23.331270 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.331184 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 03:00:23.364912 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.364264 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 03:00:23.365116 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.364583 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" containerID="cri-o://ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9" gracePeriod=30 Sep 10 03:00:23.390917 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.390876 2583 generic.go:356] "Generic (PLEG): container finished" podID="94510e4d-5379-4215-8728-1778f47979cc" containerID="a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97" exitCode=0 Sep 10 03:00:23.391135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.390920 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" Sep 10 03:00:23.391135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.390960 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" event={"ID":"94510e4d-5379-4215-8728-1778f47979cc","Type":"ContainerDied","Data":"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97"} Sep 10 03:00:23.391135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.391029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx" event={"ID":"94510e4d-5379-4215-8728-1778f47979cc","Type":"ContainerDied","Data":"8b5d120abdc3a5dcab11d1d6b4e7170786033ec898f7d96376056f8a30a2b89b"} Sep 10 03:00:23.391135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.391061 2583 scope.go:117] "RemoveContainer" containerID="a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97" Sep 10 03:00:23.403338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403244 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5mh2\" (UniqueName: \"kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2\") pod \"94510e4d-5379-4215-8728-1778f47979cc\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " Sep 10 03:00:23.403338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403311 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs\") pod \"94510e4d-5379-4215-8728-1778f47979cc\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " Sep 10 03:00:23.403338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403315 2583 scope.go:117] "RemoveContainer" containerID="a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97" Sep 10 03:00:23.403516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403342 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca\") pod \"94510e4d-5379-4215-8728-1778f47979cc\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " Sep 10 03:00:23.403516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403419 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") pod \"94510e4d-5379-4215-8728-1778f47979cc\" (UID: \"94510e4d-5379-4215-8728-1778f47979cc\") " Sep 10 03:00:23.403753 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:23.403720 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97\": container with ID starting with a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97 not found: ID does not exist" containerID="a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97" Sep 10 03:00:23.403845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.403770 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97"} err="failed to get container status \"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97\": rpc error: code = NotFound desc = could not find container \"a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97\": container with ID starting with a58cf08561ccbf9976567779a584dbe99dc5194ebc859dccba836d9bbe96df97 not found: ID does not exist" Sep 10 03:00:23.404185 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.404154 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94510e4d-5379-4215-8728-1778f47979cc" (UID: "94510e4d-5379-4215-8728-1778f47979cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:00:23.405783 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.405746 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94510e4d-5379-4215-8728-1778f47979cc" (UID: "94510e4d-5379-4215-8728-1778f47979cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:00:23.406078 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.406055 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94510e4d-5379-4215-8728-1778f47979cc" (UID: "94510e4d-5379-4215-8728-1778f47979cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:00:23.406165 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.406060 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2" (OuterVolumeSpecName: "kube-api-access-b5mh2") pod "94510e4d-5379-4215-8728-1778f47979cc" (UID: "94510e4d-5379-4215-8728-1778f47979cc"). InnerVolumeSpecName "kube-api-access-b5mh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:00:23.505548 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.505438 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:23.505548 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.505478 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mh2\" (UniqueName: \"kubernetes.io/projected/94510e4d-5379-4215-8728-1778f47979cc-kube-api-access-b5mh2\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:23.505548 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.505522 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94510e4d-5379-4215-8728-1778f47979cc-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:23.505548 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.505533 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94510e4d-5379-4215-8728-1778f47979cc-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:23.711388 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.711334 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 03:00:23.713722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:23.713694 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-387ddcaa-68ccd7bc8f-g6stx"] Sep 10 03:00:25.001100 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:25.001059 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94510e4d-5379-4215-8728-1778f47979cc" path="/var/lib/kubelet/pods/94510e4d-5379-4215-8728-1778f47979cc/volumes" Sep 10 03:00:27.941453 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.941409 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:00:27.942189 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.942162 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94510e4d-5379-4215-8728-1778f47979cc" containerName="maas-api" Sep 10 03:00:27.942189 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.942190 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94510e4d-5379-4215-8728-1778f47979cc" containerName="maas-api" Sep 10 03:00:27.942348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.942307 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94510e4d-5379-4215-8728-1778f47979cc" containerName="maas-api" Sep 10 03:00:27.946966 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.946944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:27.949680 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.949656 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-30dae5eb-serving-cert\"" Sep 10 03:00:27.958485 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:27.958461 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:00:28.053639 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.053596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.053838 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.053696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dq7\" (UniqueName: \"kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.053838 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.053742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.054007 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.053878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.154991 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.154956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dq7\" (UniqueName: \"kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.155210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.155003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.155210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.155042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.155210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.155100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.155416 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:28.155222 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-serving-cert: secret "maas-api-e2e-ait-30dae5eb-serving-cert" not found Sep 10 03:00:28.155416 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:28.155279 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls podName:4ac6c81f-990f-4a2b-933a-1328bd86c47d nodeName:}" failed. No retries permitted until 2026-09-10 03:00:28.655262965 +0000 UTC m=+832.179119630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls") pod "maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" (UID: "4ac6c81f-990f-4a2b-933a-1328bd86c47d") : secret "maas-api-e2e-ait-30dae5eb-serving-cert" not found Sep 10 03:00:28.155843 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.155816 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.157793 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.157767 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.165363 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.165343 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dq7\" (UniqueName: \"kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.661398 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.661362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.664118 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.664091 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") pod \"maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.723273 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.723243 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:28.860053 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.859962 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:28.864090 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864062 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864147 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864115 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864188 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864158 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75flh\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864226 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864196 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864231 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864371 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864352 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864457 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864418 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864457 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864443 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864482 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert\") pod \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\" (UID: \"0b0c537f-8805-49d2-865b-1f2f5539f7f2\") " Sep 10 03:00:28.864576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864488 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:00:28.864576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864524 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:00:28.864741 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864672 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:00:28.864810 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864763 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:00:28.864924 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864897 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.864924 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864922 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.865052 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864935 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.865052 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.864949 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.865159 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.865082 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:00:28.866685 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.866656 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:00:28.866685 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.866670 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh" (OuterVolumeSpecName: "kube-api-access-75flh") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "kube-api-access-75flh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:00:28.866879 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.866747 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:00:28.867084 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.867061 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b0c537f-8805-49d2-865b-1f2f5539f7f2" (UID: "0b0c537f-8805-49d2-865b-1f2f5539f7f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:00:28.966030 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.965996 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.966030 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.966027 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.966030 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.966038 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.966464 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.966046 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c537f-8805-49d2-865b-1f2f5539f7f2-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:28.966464 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:28.966056 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75flh\" (UniqueName: \"kubernetes.io/projected/0b0c537f-8805-49d2-865b-1f2f5539f7f2-kube-api-access-75flh\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:29.140171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.140140 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:00:29.142136 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:00:29.142108 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac6c81f_990f_4a2b_933a_1328bd86c47d.slice/crio-c34637b80695ed30bf6db7f4327013d349f231c7540a042b26d6ffc384955aee WatchSource:0}: Error finding container c34637b80695ed30bf6db7f4327013d349f231c7540a042b26d6ffc384955aee: Status 404 returned error can't find the container with id c34637b80695ed30bf6db7f4327013d349f231c7540a042b26d6ffc384955aee Sep 10 03:00:29.426898 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.426802 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerID="ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9" exitCode=0 Sep 10 03:00:29.426898 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.426872 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" Sep 10 03:00:29.426898 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.426886 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerDied","Data":"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9"} Sep 10 03:00:29.427189 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.426944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5" event={"ID":"0b0c537f-8805-49d2-865b-1f2f5539f7f2","Type":"ContainerDied","Data":"d729211a3a9d254e4ce29f521f4424c7d9fa0a4769a3ded10c93b166f61224d2"} Sep 10 03:00:29.427189 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.426974 2583 scope.go:117] "RemoveContainer" containerID="ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9" Sep 10 03:00:29.428309 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.428279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" event={"ID":"4ac6c81f-990f-4a2b-933a-1328bd86c47d","Type":"ContainerStarted","Data":"c34637b80695ed30bf6db7f4327013d349f231c7540a042b26d6ffc384955aee"} Sep 10 03:00:29.436089 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.436063 2583 scope.go:117] "RemoveContainer" containerID="58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5" Sep 10 03:00:29.442316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.442282 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 03:00:29.446143 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.446056 2583 scope.go:117] "RemoveContainer" containerID="ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9" Sep 10 03:00:29.446525 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:29.446505 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9\": container with ID starting with ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9 not found: ID does not exist" containerID="ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9" Sep 10 03:00:29.446604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.446533 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9"} err="failed to get container status \"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9\": rpc error: code = NotFound desc = could not find container \"ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9\": container with ID starting with ad854526ade3c41c67db85b61012b5d87dd7e580b388f6e872acb058cde4b1e9 not found: ID does not exist" Sep 10 03:00:29.446604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.446552 2583 scope.go:117] "RemoveContainer" containerID="58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5" Sep 10 03:00:29.446989 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:29.446965 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5\": container with ID starting with 58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5 not found: ID does not exist" containerID="58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5" Sep 10 03:00:29.447050 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.446998 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5"} err="failed to get container status \"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5\": rpc error: code = NotFound desc = could not find container \"58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5\": container with ID starting with 58d0f4e162d5b853f23dbb9bf9c9de4a1e400013ae17720a7fe8c9c8a67fa1b5 not found: ID does not exist" Sep 10 03:00:29.449160 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:29.449139 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-387ddcaa-gw-openshift-default-85f5bbcbb8-hs8h5"] Sep 10 03:00:30.436750 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:30.436711 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" event={"ID":"4ac6c81f-990f-4a2b-933a-1328bd86c47d","Type":"ContainerStarted","Data":"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4"} Sep 10 03:00:30.437222 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:30.436840 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:30.460024 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:30.459955 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" podStartSLOduration=2.706012524 podStartE2EDuration="3.45993386s" totalImagesPullingTime="753.921336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:30.459079807 +0000 UTC m=+833.982936495" watchObservedRunningTime="2026-09-10 03:00:30.45993386 +0000 UTC m=+833.983790549" Sep 10 03:00:31.000155 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:31.000126 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" path="/var/lib/kubelet/pods/0b0c537f-8805-49d2-865b-1f2f5539f7f2/volumes" Sep 10 03:00:36.450631 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:36.450600 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:58.558407 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.558366 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:00:58.559060 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.558631 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" podUID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" containerName="maas-api" containerID="cri-o://27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4" gracePeriod=30 Sep 10 03:00:58.820572 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.820499 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:58.871000 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.870968 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") pod \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " Sep 10 03:00:58.871195 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.871035 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2dq7\" (UniqueName: \"kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7\") pod \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " Sep 10 03:00:58.871195 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.871068 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca\") pod \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " Sep 10 03:00:58.871195 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.871083 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs\") pod \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\" (UID: \"4ac6c81f-990f-4a2b-933a-1328bd86c47d\") " Sep 10 03:00:58.871529 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.871502 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ac6c81f-990f-4a2b-933a-1328bd86c47d" (UID: "4ac6c81f-990f-4a2b-933a-1328bd86c47d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:00:58.873246 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.873215 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ac6c81f-990f-4a2b-933a-1328bd86c47d" (UID: "4ac6c81f-990f-4a2b-933a-1328bd86c47d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:00:58.873375 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.873290 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ac6c81f-990f-4a2b-933a-1328bd86c47d" (UID: "4ac6c81f-990f-4a2b-933a-1328bd86c47d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:00:58.873375 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.873360 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7" (OuterVolumeSpecName: "kube-api-access-t2dq7") pod "4ac6c81f-990f-4a2b-933a-1328bd86c47d" (UID: "4ac6c81f-990f-4a2b-933a-1328bd86c47d"). InnerVolumeSpecName "kube-api-access-t2dq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:00:58.972524 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.972463 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:58.972524 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.972521 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dq7\" (UniqueName: \"kubernetes.io/projected/4ac6c81f-990f-4a2b-933a-1328bd86c47d-kube-api-access-t2dq7\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:58.972813 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.972539 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac6c81f-990f-4a2b-933a-1328bd86c47d-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:58.972813 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:58.972554 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac6c81f-990f-4a2b-933a-1328bd86c47d-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:00:59.600020 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.599979 2583 generic.go:356] "Generic (PLEG): container finished" podID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" containerID="27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4" exitCode=0 Sep 10 03:00:59.600465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.600078 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" Sep 10 03:00:59.600465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.600123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" event={"ID":"4ac6c81f-990f-4a2b-933a-1328bd86c47d","Type":"ContainerDied","Data":"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4"} Sep 10 03:00:59.600465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.600187 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv" event={"ID":"4ac6c81f-990f-4a2b-933a-1328bd86c47d","Type":"ContainerDied","Data":"c34637b80695ed30bf6db7f4327013d349f231c7540a042b26d6ffc384955aee"} Sep 10 03:00:59.600465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.600223 2583 scope.go:117] "RemoveContainer" containerID="27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4" Sep 10 03:00:59.612585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.612553 2583 scope.go:117] "RemoveContainer" containerID="27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4" Sep 10 03:00:59.613018 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:00:59.612992 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4\": container with ID starting with 27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4 not found: ID does not exist" containerID="27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4" Sep 10 03:00:59.613116 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.613029 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4"} err="failed to get container status \"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4\": rpc error: code = NotFound desc = could not find container \"27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4\": container with ID starting with 27c51eb0f69143241c6afbc604b9551e7c6a10995cd2b2491e26bbc8e0a691d4 not found: ID does not exist" Sep 10 03:00:59.617181 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.617149 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:00:59.623417 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:00:59.623391 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30dae5eb-5bb84bddcb-9wwkv"] Sep 10 03:01:00.761549 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.761508 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:00.762082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762059 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:00.762161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762085 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:00.762161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762097 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" containerName="maas-api" Sep 10 03:01:00.762161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762105 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" containerName="maas-api" Sep 10 03:01:00.762161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762133 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:00.762161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762141 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:00.762391 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762218 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" containerName="maas-api" Sep 10 03:01:00.762391 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.762228 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:00.766723 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.766701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.769457 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.769427 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-08482c4c-openshift-default-dockercfg-dspwq\"" Sep 10 03:01:00.777602 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.777569 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:00.891365 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891324 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891365 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891375 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891620 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891620 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5pc\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891736 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891613 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891736 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891677 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891736 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891886 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.891886 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.891823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993328 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993531 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993531 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993531 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993514 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993711 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5pc\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993711 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993602 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.993880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994033 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994011 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994112 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.993944 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994112 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994220 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994220 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994220 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.994423 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.994398 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.995900 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.995846 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:00.996445 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:00.996425 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:01.000498 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.000469 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac6c81f-990f-4a2b-933a-1328bd86c47d" path="/var/lib/kubelet/pods/4ac6c81f-990f-4a2b-933a-1328bd86c47d/volumes" Sep 10 03:01:01.002975 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.002947 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:01.002975 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.002970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5pc\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc\") pod \"e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:01.080714 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.080622 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:01.355458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.355430 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:01.357540 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:01:01.357508 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582bcb4c_2bbf_4f0b_a7c7_c7955f6d827f.slice/crio-98a895fde4690b3324f651e0854898b1f183cb2f0438b20bd5ba7351315644cb WatchSource:0}: Error finding container 98a895fde4690b3324f651e0854898b1f183cb2f0438b20bd5ba7351315644cb: Status 404 returned error can't find the container with id 98a895fde4690b3324f651e0854898b1f183cb2f0438b20bd5ba7351315644cb Sep 10 03:01:01.359782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.359745 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:01.359886 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.359818 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:01.359886 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.359848 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:01.614151 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.614054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerStarted","Data":"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f"} Sep 10 03:01:01.614151 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.614127 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerStarted","Data":"98a895fde4690b3324f651e0854898b1f183cb2f0438b20bd5ba7351315644cb"} Sep 10 03:01:01.635535 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:01.635476 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podStartSLOduration=1.6354571039999999 podStartE2EDuration="1.635457104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:01.633226124 +0000 UTC m=+865.157082813" watchObservedRunningTime="2026-09-10 03:01:01.635457104 +0000 UTC m=+865.159313791" Sep 10 03:01:02.081074 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:02.081036 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:02.086045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:02.086017 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:02.620521 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:02.620485 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:02.621305 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:02.621288 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:04.111625 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.111587 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:04.112234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.112212 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c537f-8805-49d2-865b-1f2f5539f7f2" containerName="istio-proxy" Sep 10 03:01:04.115395 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.115375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.117718 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.117697 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-08482c4c-serving-cert\"" Sep 10 03:01:04.128002 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.127974 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:04.229815 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.229754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.230119 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.230068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.230346 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.230314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.230624 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.230600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6dj\" (UniqueName: \"kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.331678 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.331634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6dj\" (UniqueName: \"kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.331918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.331766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.331918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.331798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.331918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.331837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.332124 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:04.331950 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-serving-cert: secret "maas-api-e2e-derive-08482c4c-serving-cert" not found Sep 10 03:01:04.332124 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:04.332026 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls podName:cf16a213-da43-45df-88f8-e3d80b17ec6c nodeName:}" failed. No retries permitted until 2026-09-10 03:01:04.832010679 +0000 UTC m=+868.355867343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls") pod "maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" (UID: "cf16a213-da43-45df-88f8-e3d80b17ec6c") : secret "maas-api-e2e-derive-08482c4c-serving-cert" not found Sep 10 03:01:04.332605 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.332581 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.334294 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.334274 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.340338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.340311 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6dj\" (UniqueName: \"kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.839303 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.839263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:04.842395 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:04.842361 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") pod \"maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:05.029929 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:05.029883 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:05.307271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:05.307234 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:05.309337 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:01:05.309300 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf16a213_da43_45df_88f8_e3d80b17ec6c.slice/crio-7f304fd7aefae0acb5caf604ff6f9f527a8dfb838cbf7af62eec98ad40fb0d5c WatchSource:0}: Error finding container 7f304fd7aefae0acb5caf604ff6f9f527a8dfb838cbf7af62eec98ad40fb0d5c: Status 404 returned error can't find the container with id 7f304fd7aefae0acb5caf604ff6f9f527a8dfb838cbf7af62eec98ad40fb0d5c Sep 10 03:01:05.641847 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:05.641753 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" event={"ID":"cf16a213-da43-45df-88f8-e3d80b17ec6c","Type":"ContainerStarted","Data":"7f304fd7aefae0acb5caf604ff6f9f527a8dfb838cbf7af62eec98ad40fb0d5c"} Sep 10 03:01:06.649627 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:06.649589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" event={"ID":"cf16a213-da43-45df-88f8-e3d80b17ec6c","Type":"ContainerStarted","Data":"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc"} Sep 10 03:01:06.650094 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:06.649684 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:06.667580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:06.667525 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" podStartSLOduration=2.048213645 podStartE2EDuration="2.667508359s" totalImagesPullingTime="619.294714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:06.665592247 +0000 UTC m=+870.189448934" watchObservedRunningTime="2026-09-10 03:01:06.667508359 +0000 UTC m=+870.191365046" Sep 10 03:01:12.661545 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:12.661513 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:25.761742 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.761692 2583 generic.go:356] "Generic (PLEG): container finished" podID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerID="1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f" exitCode=0 Sep 10 03:01:25.762344 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.761828 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerDied","Data":"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f"} Sep 10 03:01:25.762500 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.762474 2583 scope.go:117] "RemoveContainer" containerID="1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f" Sep 10 03:01:25.764202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.764095 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:25.764202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.764196 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:25.764389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:25.764249 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:01:26.769457 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:26.769422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerStarted","Data":"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97"} Sep 10 03:01:27.081698 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:27.081588 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:27.081889 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:27.081809 2583 patch_prober.go:28] interesting pod/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 03:01:27.081943 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:27.081889 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 03:01:28.081343 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:28.081305 2583 patch_prober.go:28] interesting pod/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 03:01:28.081762 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:28.081382 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 03:01:29.081582 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:29.081492 2583 patch_prober.go:28] interesting pod/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 03:01:29.081582 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:29.081557 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 03:01:30.081795 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:30.081760 2583 patch_prober.go:28] interesting pod/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 03:01:30.082295 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:30.081816 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 03:01:31.081970 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.081930 2583 patch_prober.go:28] interesting pod/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 10 03:01:31.082379 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.081989 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 10 03:01:31.275631 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.275592 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 03:01:31.276033 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.276002 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" podUID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" containerName="maas-api" containerID="cri-o://f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8" gracePeriod=30 Sep 10 03:01:31.629289 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.629258 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 03:01:31.744110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744020 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") pod \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " Sep 10 03:01:31.744110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744061 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzjbp\" (UniqueName: \"kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp\") pod \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " Sep 10 03:01:31.744357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744194 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs\") pod \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " Sep 10 03:01:31.744357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744259 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca\") pod \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\" (UID: \"8b3783d7-6800-417c-95ed-bc7ec1bdd0db\") " Sep 10 03:01:31.744649 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744618 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b3783d7-6800-417c-95ed-bc7ec1bdd0db" (UID: "8b3783d7-6800-417c-95ed-bc7ec1bdd0db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:01:31.744785 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.744725 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:31.746202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.746181 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b3783d7-6800-417c-95ed-bc7ec1bdd0db" (UID: "8b3783d7-6800-417c-95ed-bc7ec1bdd0db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:01:31.746301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.746259 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp" (OuterVolumeSpecName: "kube-api-access-wzjbp") pod "8b3783d7-6800-417c-95ed-bc7ec1bdd0db" (UID: "8b3783d7-6800-417c-95ed-bc7ec1bdd0db"). InnerVolumeSpecName "kube-api-access-wzjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:01:31.746301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.746286 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b3783d7-6800-417c-95ed-bc7ec1bdd0db" (UID: "8b3783d7-6800-417c-95ed-bc7ec1bdd0db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:01:31.817242 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.817204 2583 generic.go:356] "Generic (PLEG): container finished" podID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" containerID="f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8" exitCode=0 Sep 10 03:01:31.817242 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.817247 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" Sep 10 03:01:31.817510 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.817285 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" event={"ID":"8b3783d7-6800-417c-95ed-bc7ec1bdd0db","Type":"ContainerDied","Data":"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8"} Sep 10 03:01:31.817510 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.817336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl" event={"ID":"8b3783d7-6800-417c-95ed-bc7ec1bdd0db","Type":"ContainerDied","Data":"3f7e959c700e93e51594fe9c09627fa06d5f51c8d3d720b8ba877c401b2d7127"} Sep 10 03:01:31.817510 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.817359 2583 scope.go:117] "RemoveContainer" containerID="f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8" Sep 10 03:01:31.828887 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.828841 2583 scope.go:117] "RemoveContainer" containerID="f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8" Sep 10 03:01:31.829177 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:31.829158 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8\": container with ID starting with f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8 not found: ID does not exist" containerID="f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8" Sep 10 03:01:31.829235 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.829187 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8"} err="failed to get container status \"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8\": rpc error: code = NotFound desc = could not find container \"f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8\": container with ID starting with f8f004ba4e9bb41e440ad2f88896ada4346b0228a16a762679dbe6189f3509b8 not found: ID does not exist" Sep 10 03:01:31.837739 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.837717 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 03:01:31.842143 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.842117 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5ab28b-796498876b-84qtl"] Sep 10 03:01:31.846078 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.846060 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:31.846144 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.846081 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzjbp\" (UniqueName: \"kubernetes.io/projected/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-kube-api-access-wzjbp\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:31.846144 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:31.846091 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3783d7-6800-417c-95ed-bc7ec1bdd0db-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:32.085154 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:32.085123 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:32.085535 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:32.085399 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:32.086167 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:32.086150 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:33.000275 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:33.000240 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" path="/var/lib/kubelet/pods/8b3783d7-6800-417c-95ed-bc7ec1bdd0db/volumes" Sep 10 03:01:34.823941 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:34.823903 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:34.824426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:34.824161 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" podUID="cf16a213-da43-45df-88f8-e3d80b17ec6c" containerName="maas-api" containerID="cri-o://286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc" gracePeriod=30 Sep 10 03:01:35.087209 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.087137 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:35.173905 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.173814 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:35.174837 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.174243 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" containerID="cri-o://1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97" gracePeriod=30 Sep 10 03:01:35.179902 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.179874 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs\") pod \"cf16a213-da43-45df-88f8-e3d80b17ec6c\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " Sep 10 03:01:35.180053 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.179915 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") pod \"cf16a213-da43-45df-88f8-e3d80b17ec6c\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " Sep 10 03:01:35.180420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.180094 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca\") pod \"cf16a213-da43-45df-88f8-e3d80b17ec6c\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " Sep 10 03:01:35.180420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.180126 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z6dj\" (UniqueName: \"kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj\") pod \"cf16a213-da43-45df-88f8-e3d80b17ec6c\" (UID: \"cf16a213-da43-45df-88f8-e3d80b17ec6c\") " Sep 10 03:01:35.181056 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.181003 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf16a213-da43-45df-88f8-e3d80b17ec6c" (UID: "cf16a213-da43-45df-88f8-e3d80b17ec6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:01:35.183158 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.183130 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf16a213-da43-45df-88f8-e3d80b17ec6c" (UID: "cf16a213-da43-45df-88f8-e3d80b17ec6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:01:35.184612 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.184579 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf16a213-da43-45df-88f8-e3d80b17ec6c" (UID: "cf16a213-da43-45df-88f8-e3d80b17ec6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:01:35.184721 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.184704 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj" (OuterVolumeSpecName: "kube-api-access-8z6dj") pod "cf16a213-da43-45df-88f8-e3d80b17ec6c" (UID: "cf16a213-da43-45df-88f8-e3d80b17ec6c"). InnerVolumeSpecName "kube-api-access-8z6dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:01:35.282084 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.282037 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf16a213-da43-45df-88f8-e3d80b17ec6c-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:35.282084 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.282082 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6dj\" (UniqueName: \"kubernetes.io/projected/cf16a213-da43-45df-88f8-e3d80b17ec6c-kube-api-access-8z6dj\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:35.282361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.282098 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:35.282361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.282110 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf16a213-da43-45df-88f8-e3d80b17ec6c-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:35.846388 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.846348 2583 generic.go:356] "Generic (PLEG): container finished" podID="cf16a213-da43-45df-88f8-e3d80b17ec6c" containerID="286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc" exitCode=0 Sep 10 03:01:35.846789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.846396 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" Sep 10 03:01:35.846789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.846420 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" event={"ID":"cf16a213-da43-45df-88f8-e3d80b17ec6c","Type":"ContainerDied","Data":"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc"} Sep 10 03:01:35.846789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.846474 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz" event={"ID":"cf16a213-da43-45df-88f8-e3d80b17ec6c","Type":"ContainerDied","Data":"7f304fd7aefae0acb5caf604ff6f9f527a8dfb838cbf7af62eec98ad40fb0d5c"} Sep 10 03:01:35.846789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.846496 2583 scope.go:117] "RemoveContainer" containerID="286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc" Sep 10 03:01:35.856851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.856832 2583 scope.go:117] "RemoveContainer" containerID="286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc" Sep 10 03:01:35.857211 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:35.857192 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc\": container with ID starting with 286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc not found: ID does not exist" containerID="286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc" Sep 10 03:01:35.857263 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.857221 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc"} err="failed to get container status \"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc\": rpc error: code = NotFound desc = could not find container \"286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc\": container with ID starting with 286c5e2b97d78d619765c2be2ed8d9e4b18b97f6cd42d2af2dd929e567c1f5bc not found: ID does not exist" Sep 10 03:01:35.880330 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.880266 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:35.882799 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:35.882773 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08482c4c-659bf786cb-g9ljz"] Sep 10 03:01:37.002751 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:37.002722 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:01:37.003601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:37.003567 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf16a213-da43-45df-88f8-e3d80b17ec6c" path="/var/lib/kubelet/pods/cf16a213-da43-45df-88f8-e3d80b17ec6c/volumes" Sep 10 03:01:37.005750 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:37.005729 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:01:40.539528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.539499 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:40.638435 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638338 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638435 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638399 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks5pc\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638443 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638464 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638620 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638814 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638674 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638814 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638741 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data" (OuterVolumeSpecName: "istio-data") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:01:40.638814 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638754 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.638814 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638800 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:01:40.639087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.639087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.638920 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket\") pod \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\" (UID: \"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f\") " Sep 10 03:01:40.639087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639073 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:01:40.639238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639126 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:01:40.639238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639225 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:01:40.639511 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639490 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.639601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639514 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.639601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639530 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.639601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639543 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.639601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.639555 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.641287 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.641260 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:01:40.641287 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.641264 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:01:40.641420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.641341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token" (OuterVolumeSpecName: "istio-token") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:01:40.641420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.641369 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc" (OuterVolumeSpecName: "kube-api-access-ks5pc") pod "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" (UID: "582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f"). InnerVolumeSpecName "kube-api-access-ks5pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:01:40.740874 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.740834 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.741046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.740894 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.741046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.740906 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5pc\" (UniqueName: \"kubernetes.io/projected/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-kube-api-access-ks5pc\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.741046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.740917 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:01:40.877798 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.877760 2583 generic.go:356] "Generic (PLEG): container finished" podID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerID="1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97" exitCode=0 Sep 10 03:01:40.877978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.877807 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" Sep 10 03:01:40.877978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.877840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerDied","Data":"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97"} Sep 10 03:01:40.877978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.877917 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp" event={"ID":"582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f","Type":"ContainerDied","Data":"98a895fde4690b3324f651e0854898b1f183cb2f0438b20bd5ba7351315644cb"} Sep 10 03:01:40.877978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.877949 2583 scope.go:117] "RemoveContainer" containerID="1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97" Sep 10 03:01:40.887719 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.887701 2583 scope.go:117] "RemoveContainer" containerID="1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f" Sep 10 03:01:40.896137 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.896115 2583 scope.go:117] "RemoveContainer" containerID="1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97" Sep 10 03:01:40.896601 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:40.896568 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97\": container with ID starting with 1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97 not found: ID does not exist" containerID="1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97" Sep 10 03:01:40.896716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.896611 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97"} err="failed to get container status \"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97\": rpc error: code = NotFound desc = could not find container \"1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97\": container with ID starting with 1aab33a2b21471b4c2d6db210640409fa1cec653f337e1883abe03c4dd4f2f97 not found: ID does not exist" Sep 10 03:01:40.896716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.896634 2583 scope.go:117] "RemoveContainer" containerID="1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f" Sep 10 03:01:40.897115 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:01:40.897075 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f\": container with ID starting with 1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f not found: ID does not exist" containerID="1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f" Sep 10 03:01:40.897115 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.897101 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f"} err="failed to get container status \"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f\": rpc error: code = NotFound desc = could not find container \"1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f\": container with ID starting with 1b36de06571154a3019efd62b4320aca85bee82ebe2f63bad405d2239833d12f not found: ID does not exist" Sep 10 03:01:40.899092 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.899068 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:40.902820 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:40.902797 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-08482c4c-openshift-default-85f494fcbd-kn5dp"] Sep 10 03:01:41.000238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:01:41.000197 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" path="/var/lib/kubelet/pods/582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f/volumes" Sep 10 03:02:14.917723 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.917686 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918091 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" containerName="maas-api" Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918103 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" containerName="maas-api" Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918117 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf16a213-da43-45df-88f8-e3d80b17ec6c" containerName="maas-api" Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918123 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf16a213-da43-45df-88f8-e3d80b17ec6c" containerName="maas-api" Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918132 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:14.918161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918138 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:14.918376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918205 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:14.918376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918213 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3783d7-6800-417c-95ed-bc7ec1bdd0db" containerName="maas-api" Sep 10 03:02:14.918376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.918222 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf16a213-da43-45df-88f8-e3d80b17ec6c" containerName="maas-api" Sep 10 03:02:14.921604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.921580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:14.924036 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.924012 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-af329f-openshift-default-dockercfg-pdnt5\"" Sep 10 03:02:14.934646 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:14.934620 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:02:15.074203 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074203 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074268 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074613 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdph6\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.074613 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.074524 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176131 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdph6\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176131 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176362 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176362 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176362 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176236 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176553 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176553 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176553 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176553 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176622 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176678 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176734 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.176943 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.176921 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.177133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.177109 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.178878 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.178841 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.179070 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.179051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.183836 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.183810 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdph6\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.184115 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.184097 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token\") pod \"e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.234274 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.234233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:15.532530 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.532481 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:02:15.534644 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:02:15.534597 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd88bcb_346a_4e6f_8ac2_a17ff3d92d22.slice/crio-660a49fb2a4a973eb6ba12cfaa71327dbc2c2f7450db42e88a5a53ef6447fec4 WatchSource:0}: Error finding container 660a49fb2a4a973eb6ba12cfaa71327dbc2c2f7450db42e88a5a53ef6447fec4: Status 404 returned error can't find the container with id 660a49fb2a4a973eb6ba12cfaa71327dbc2c2f7450db42e88a5a53ef6447fec4 Sep 10 03:02:15.537010 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.536975 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:15.537107 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.537051 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:15.537107 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:15.537097 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:16.059646 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:16.059613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerStarted","Data":"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69"} Sep 10 03:02:16.060080 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:16.059670 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerStarted","Data":"660a49fb2a4a973eb6ba12cfaa71327dbc2c2f7450db42e88a5a53ef6447fec4"} Sep 10 03:02:16.080108 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:16.080052 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podStartSLOduration=2.080028965 podStartE2EDuration="2.080028965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:16.079156965 +0000 UTC m=+939.603013643" watchObservedRunningTime="2026-09-10 03:02:16.080028965 +0000 UTC m=+939.603885652" Sep 10 03:02:16.234940 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:16.234900 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:16.240633 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:16.240602 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:17.064911 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:17.064878 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:17.065803 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:17.065784 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:19.208232 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.208202 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:02:19.208634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.208606 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:19.208634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.208618 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:19.208712 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.208692 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="582bcb4c-2bbf-4f0b-a7c7-c7955f6d827f" containerName="istio-proxy" Sep 10 03:02:19.211957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.211936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.216359 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.216336 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-af329f-serving-cert\"" Sep 10 03:02:19.242517 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.242482 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:02:19.319312 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.319274 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.319528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.319396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.319528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.319426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrph\" (UniqueName: \"kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.319528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.319454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.420599 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.420559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.420770 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.420626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.420770 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.420735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrph\" (UniqueName: \"kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.420887 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:02:19.420757 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-serving-cert: secret "maas-api-e2e-explicit-af329f-serving-cert" not found Sep 10 03:02:19.420887 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.420772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.420887 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:02:19.420836 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls podName:2dae8405-9ef5-4010-a5c6-a249869747e2 nodeName:}" failed. No retries permitted until 2026-09-10 03:02:19.920814143 +0000 UTC m=+943.444670828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls") pod "maas-api-e2e-explicit-af329f-5575b9786b-lplz6" (UID: "2dae8405-9ef5-4010-a5c6-a249869747e2") : secret "maas-api-e2e-explicit-af329f-serving-cert" not found Sep 10 03:02:19.421506 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.421479 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.423265 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.423235 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.428653 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.428630 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrph\" (UniqueName: \"kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.928902 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.928834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:19.933421 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:19.933387 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") pod \"maas-api-e2e-explicit-af329f-5575b9786b-lplz6\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:20.122796 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:20.122760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:20.398746 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:20.398718 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:02:20.401581 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:02:20.401549 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dae8405_9ef5_4010_a5c6_a249869747e2.slice/crio-2a851d58248197bf6b3b81d488af353f819a3683c9f693a41406de787b2c3b09 WatchSource:0}: Error finding container 2a851d58248197bf6b3b81d488af353f819a3683c9f693a41406de787b2c3b09: Status 404 returned error can't find the container with id 2a851d58248197bf6b3b81d488af353f819a3683c9f693a41406de787b2c3b09 Sep 10 03:02:20.403175 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:20.403150 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 03:02:21.088716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:21.088609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" event={"ID":"2dae8405-9ef5-4010-a5c6-a249869747e2","Type":"ContainerStarted","Data":"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d"} Sep 10 03:02:21.088716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:21.088672 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" event={"ID":"2dae8405-9ef5-4010-a5c6-a249869747e2","Type":"ContainerStarted","Data":"2a851d58248197bf6b3b81d488af353f819a3683c9f693a41406de787b2c3b09"} Sep 10 03:02:21.089022 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:21.088742 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:21.105405 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:21.105352 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" podStartSLOduration=1.673840546 podStartE2EDuration="2.105330413s" totalImagesPullingTime="431.489867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:21.104432338 +0000 UTC m=+944.628289022" watchObservedRunningTime="2026-09-10 03:02:21.105330413 +0000 UTC m=+944.629187100" Sep 10 03:02:27.099264 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:27.099235 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:02:41.195516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.195477 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerID="d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69" exitCode=0 Sep 10 03:02:41.195994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.195552 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerDied","Data":"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69"} Sep 10 03:02:41.195994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.195992 2583 scope.go:117] "RemoveContainer" containerID="d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69" Sep 10 03:02:41.197195 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.197155 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:41.197296 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.197230 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:41.197296 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:41.197266 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:02:42.207223 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:42.207176 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerStarted","Data":"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244"} Sep 10 03:02:42.235070 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:42.235037 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:42.236106 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:42.236076 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:42.236228 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:42.236147 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:43.235552 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:43.235509 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:43.235996 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:43.235569 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:44.235485 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:44.235450 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:44.235670 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:44.235519 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:45.235569 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:45.235524 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:45.235820 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:45.235589 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:46.235741 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:46.235699 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:46.236032 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:46.235762 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:47.065337 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:47.065299 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:47.234714 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:47.234678 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 10 03:02:47.234938 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:47.234753 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 10 03:02:48.240051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:48.240020 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:02:48.241000 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:02:48.240980 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:03:11.041332 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.041297 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:03:11.044752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.044734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.047900 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.047881 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-842ba8c2-openshift-default-dockercfg-kwvcm\"" Sep 10 03:03:11.074577 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.074543 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:03:11.171548 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97d8\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.171760 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.171760 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.171914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.171914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.171914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.172088 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.172088 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.171999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.172088 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.172020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273547 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97d8\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.273695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.273660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.274110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.274045 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.274110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.274066 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.274217 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.274145 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.274302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.274281 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.274356 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.274340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.275784 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.275759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.276177 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.276152 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.282644 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.282614 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97d8\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.282829 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.282811 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token\") pod \"e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.355320 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.355233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:11.636153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.636127 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:03:11.637990 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:03:11.637962 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026bb835_58e5_4017_a161_6a849913f261.slice/crio-f98254274d0fe1c121b8c997771e74b9a6b1708d86f6a4d9ee035df1d2d51876 WatchSource:0}: Error finding container f98254274d0fe1c121b8c997771e74b9a6b1708d86f6a4d9ee035df1d2d51876: Status 404 returned error can't find the container with id f98254274d0fe1c121b8c997771e74b9a6b1708d86f6a4d9ee035df1d2d51876 Sep 10 03:03:11.640461 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.640422 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:11.640570 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.640493 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:11.640570 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:11.640545 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:12.366626 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:12.366589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerStarted","Data":"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347"} Sep 10 03:03:12.366626 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:12.366646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerStarted","Data":"f98254274d0fe1c121b8c997771e74b9a6b1708d86f6a4d9ee035df1d2d51876"} Sep 10 03:03:12.389127 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:12.389070 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podStartSLOduration=1.389053822 podStartE2EDuration="1.389053822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:12.386122466 +0000 UTC m=+995.909979179" watchObservedRunningTime="2026-09-10 03:03:12.389053822 +0000 UTC m=+995.912910508" Sep 10 03:03:13.356032 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:13.355983 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:13.361426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:13.361399 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:13.372794 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:13.372763 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:13.373843 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:13.373823 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:18.390136 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.390098 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:03:18.394135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.394110 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.399348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.399320 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-842ba8c2-serving-cert\"" Sep 10 03:03:18.409481 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.409453 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:03:18.547030 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.546992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.547205 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.547117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.547205 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.547153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.547280 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.547235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6bc\" (UniqueName: \"kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.648390 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.648294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.648576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.648402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.648576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.648435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.648576 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:03:18.648449 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-serving-cert: secret "maas-api-e2e-mt-842ba8c2-serving-cert" not found Sep 10 03:03:18.648576 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:03:18.648531 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls podName:f8b2d053-e60c-4aae-91b4-61dfa0325c29 nodeName:}" failed. No retries permitted until 2026-09-10 03:03:19.148509528 +0000 UTC m=+1002.672366215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls") pod "maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" (UID: "f8b2d053-e60c-4aae-91b4-61dfa0325c29") : secret "maas-api-e2e-mt-842ba8c2-serving-cert" not found Sep 10 03:03:18.648817 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.648589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6bc\" (UniqueName: \"kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.649251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.649226 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.651008 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.650988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:18.658279 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:18.658242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6bc\" (UniqueName: \"kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:19.153355 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:19.153298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:19.155938 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:19.155909 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") pod \"maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:19.308089 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:19.308057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:19.594833 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:19.594801 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:03:19.596900 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:03:19.596872 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b2d053_e60c_4aae_91b4_61dfa0325c29.slice/crio-89a02e449f2de92e6b26d3247796fc2e7f46dc962460f330f758996f40928eb2 WatchSource:0}: Error finding container 89a02e449f2de92e6b26d3247796fc2e7f46dc962460f330f758996f40928eb2: Status 404 returned error can't find the container with id 89a02e449f2de92e6b26d3247796fc2e7f46dc962460f330f758996f40928eb2 Sep 10 03:03:20.423334 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:20.423299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" event={"ID":"f8b2d053-e60c-4aae-91b4-61dfa0325c29","Type":"ContainerStarted","Data":"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b"} Sep 10 03:03:20.423546 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:20.423358 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" event={"ID":"f8b2d053-e60c-4aae-91b4-61dfa0325c29","Type":"ContainerStarted","Data":"89a02e449f2de92e6b26d3247796fc2e7f46dc962460f330f758996f40928eb2"} Sep 10 03:03:20.423546 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:20.423437 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:20.445524 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:20.445406 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" podStartSLOduration=1.860541219 podStartE2EDuration="2.445389302s" totalImagesPullingTime="584.848083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:20.442271183 +0000 UTC m=+1003.966127871" watchObservedRunningTime="2026-09-10 03:03:20.445389302 +0000 UTC m=+1003.969245979" Sep 10 03:03:26.434015 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:26.433985 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:03:44.414090 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:44.414034 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 03:03:44.414576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:44.414109 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 03:03:45.564938 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.564902 2583 generic.go:356] "Generic (PLEG): container finished" podID="026bb835-58e5-4017-a161-6a849913f261" containerID="1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347" exitCode=0 Sep 10 03:03:45.565369 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.564979 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerDied","Data":"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347"} Sep 10 03:03:45.565434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.565407 2583 scope.go:117] "RemoveContainer" containerID="1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347" Sep 10 03:03:45.566651 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.566621 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:45.566715 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.566700 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:45.566753 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:45.566736 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:03:46.572409 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:46.572374 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerStarted","Data":"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1"} Sep 10 03:03:47.355760 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:47.355718 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:47.355999 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:47.355940 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 10 03:03:47.356059 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:47.355989 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 10 03:03:48.355989 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:48.355946 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 10 03:03:48.356399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:48.356022 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 10 03:03:49.355964 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:49.355924 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 10 03:03:49.356225 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:49.356008 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 10 03:03:50.356344 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:50.356309 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 10 03:03:50.356905 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:50.356367 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 10 03:03:51.356168 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:51.356128 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 10 03:03:51.356364 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:51.356207 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 10 03:03:52.359745 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:52.359712 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:52.360171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:52.360075 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:52.360934 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:52.360916 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:03:59.980178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:59.980138 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:03:59.980948 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:03:59.980911 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" podUID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" containerName="maas-api" containerID="cri-o://b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b" gracePeriod=30 Sep 10 03:04:00.236120 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.236053 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:04:00.384882 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.383949 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") pod \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " Sep 10 03:04:00.384882 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.384056 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca\") pod \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " Sep 10 03:04:00.384882 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.384102 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr6bc\" (UniqueName: \"kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc\") pod \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " Sep 10 03:04:00.384882 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.384181 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs\") pod \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\" (UID: \"f8b2d053-e60c-4aae-91b4-61dfa0325c29\") " Sep 10 03:04:00.394402 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.394363 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8b2d053-e60c-4aae-91b4-61dfa0325c29" (UID: "f8b2d053-e60c-4aae-91b4-61dfa0325c29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:04:00.395094 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.395047 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8b2d053-e60c-4aae-91b4-61dfa0325c29" (UID: "f8b2d053-e60c-4aae-91b4-61dfa0325c29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:04:00.397094 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.397047 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc" (OuterVolumeSpecName: "kube-api-access-vr6bc") pod "f8b2d053-e60c-4aae-91b4-61dfa0325c29" (UID: "f8b2d053-e60c-4aae-91b4-61dfa0325c29"). InnerVolumeSpecName "kube-api-access-vr6bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:04:00.397227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.397091 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f8b2d053-e60c-4aae-91b4-61dfa0325c29" (UID: "f8b2d053-e60c-4aae-91b4-61dfa0325c29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:04:00.486234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.486131 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2d053-e60c-4aae-91b4-61dfa0325c29-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:00.486234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.486169 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6bc\" (UniqueName: \"kubernetes.io/projected/f8b2d053-e60c-4aae-91b4-61dfa0325c29-kube-api-access-vr6bc\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:00.486234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.486179 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:00.486234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.486187 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b2d053-e60c-4aae-91b4-61dfa0325c29-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:00.657757 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.657723 2583 generic.go:356] "Generic (PLEG): container finished" podID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" containerID="b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b" exitCode=0 Sep 10 03:04:00.657955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.657767 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" Sep 10 03:04:00.657955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.657808 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" event={"ID":"f8b2d053-e60c-4aae-91b4-61dfa0325c29","Type":"ContainerDied","Data":"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b"} Sep 10 03:04:00.657955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.657891 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh" event={"ID":"f8b2d053-e60c-4aae-91b4-61dfa0325c29","Type":"ContainerDied","Data":"89a02e449f2de92e6b26d3247796fc2e7f46dc962460f330f758996f40928eb2"} Sep 10 03:04:00.657955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.657923 2583 scope.go:117] "RemoveContainer" containerID="b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b" Sep 10 03:04:00.667049 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.667005 2583 scope.go:117] "RemoveContainer" containerID="b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b" Sep 10 03:04:00.667328 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:04:00.667310 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b\": container with ID starting with b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b not found: ID does not exist" containerID="b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b" Sep 10 03:04:00.667375 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.667336 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b"} err="failed to get container status \"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b\": rpc error: code = NotFound desc = could not find container \"b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b\": container with ID starting with b458c2a7101ea65c483d4fb4782064d701e041480cce696ff9919f66b888302b not found: ID does not exist" Sep 10 03:04:00.678166 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.678142 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:04:00.680253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:00.680233 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-842ba8c2-5c44f8d8b8-7nsxh"] Sep 10 03:04:01.001130 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:01.001088 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" path="/var/lib/kubelet/pods/f8b2d053-e60c-4aae-91b4-61dfa0325c29/volumes" Sep 10 03:04:06.970129 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:06.970097 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:04:06.970581 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:06.970334 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" containerID="cri-o://cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1" gracePeriod=30 Sep 10 03:04:07.362920 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:07.362872 2583 patch_prober.go:28] interesting pod/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 03:04:07.363091 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:07.362933 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 03:04:12.225313 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.225285 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:04:12.307604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307567 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.307784 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307701 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.307784 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307766 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.307927 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307792 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.307927 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307846 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j97d8\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.307927 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307906 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.308087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.307943 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:04:12.308087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308030 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:04:12.308087 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308080 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.308253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308157 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.308253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308178 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:04:12.308253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308205 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket\") pod \"026bb835-58e5-4017-a161-6a849913f261\" (UID: \"026bb835-58e5-4017-a161-6a849913f261\") " Sep 10 03:04:12.308418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308354 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:04:12.308418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308379 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data" (OuterVolumeSpecName: "istio-data") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:04:12.308644 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308626 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.308722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308648 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.308722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308665 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.308722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308681 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.308722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.308695 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026bb835-58e5-4017-a161-6a849913f261-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.310183 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.310160 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token" (OuterVolumeSpecName: "istio-token") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:04:12.310260 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.310160 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8" (OuterVolumeSpecName: "kube-api-access-j97d8") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "kube-api-access-j97d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:04:12.310356 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.310341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:04:12.310474 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.310459 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "026bb835-58e5-4017-a161-6a849913f261" (UID: "026bb835-58e5-4017-a161-6a849913f261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:04:12.409871 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.409827 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026bb835-58e5-4017-a161-6a849913f261-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.410044 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.409885 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97d8\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-kube-api-access-j97d8\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.410044 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.409896 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026bb835-58e5-4017-a161-6a849913f261-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.410044 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.409904 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026bb835-58e5-4017-a161-6a849913f261-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:12.726951 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.726839 2583 generic.go:356] "Generic (PLEG): container finished" podID="026bb835-58e5-4017-a161-6a849913f261" containerID="cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1" exitCode=0 Sep 10 03:04:12.726951 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.726902 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" Sep 10 03:04:12.727194 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.726900 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerDied","Data":"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1"} Sep 10 03:04:12.727194 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.727033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6" event={"ID":"026bb835-58e5-4017-a161-6a849913f261","Type":"ContainerDied","Data":"f98254274d0fe1c121b8c997771e74b9a6b1708d86f6a4d9ee035df1d2d51876"} Sep 10 03:04:12.727194 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.727063 2583 scope.go:117] "RemoveContainer" containerID="cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1" Sep 10 03:04:12.737451 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.737431 2583 scope.go:117] "RemoveContainer" containerID="1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347" Sep 10 03:04:12.746796 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.746771 2583 scope.go:117] "RemoveContainer" containerID="cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1" Sep 10 03:04:12.747559 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:04:12.747453 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1\": container with ID starting with cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1 not found: ID does not exist" containerID="cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1" Sep 10 03:04:12.747559 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.747500 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1"} err="failed to get container status \"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1\": rpc error: code = NotFound desc = could not find container \"cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1\": container with ID starting with cfcd499c2a4f0c7f7f0dd32b571346e86879d2bfd1217bc2ebd7852d183c55f1 not found: ID does not exist" Sep 10 03:04:12.747559 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.747526 2583 scope.go:117] "RemoveContainer" containerID="1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347" Sep 10 03:04:12.747964 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:04:12.747846 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347\": container with ID starting with 1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347 not found: ID does not exist" containerID="1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347" Sep 10 03:04:12.748017 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.747977 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347"} err="failed to get container status \"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347\": rpc error: code = NotFound desc = could not find container \"1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347\": container with ID starting with 1d8162099501bbfb8ba2118217b845c6c28fc45d313512f16d73daa7e8a24347 not found: ID does not exist" Sep 10 03:04:12.748784 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.748767 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:04:12.750848 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:12.750828 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-842ba8c2-openshift-default-557f5fc9d4-pf9d6"] Sep 10 03:04:13.000663 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:13.000633 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026bb835-58e5-4017-a161-6a849913f261" path="/var/lib/kubelet/pods/026bb835-58e5-4017-a161-6a849913f261/volumes" Sep 10 03:04:15.840915 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:15.840871 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:04:15.841295 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:15.841114 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" podUID="2dae8405-9ef5-4010-a5c6-a249869747e2" containerName="maas-api" containerID="cri-o://42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d" gracePeriod=30 Sep 10 03:04:16.104331 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.104230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:04:16.147354 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147311 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca\") pod \"2dae8405-9ef5-4010-a5c6-a249869747e2\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " Sep 10 03:04:16.147552 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147472 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs\") pod \"2dae8405-9ef5-4010-a5c6-a249869747e2\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " Sep 10 03:04:16.147552 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147538 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvrph\" (UniqueName: \"kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph\") pod \"2dae8405-9ef5-4010-a5c6-a249869747e2\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " Sep 10 03:04:16.147679 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147591 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") pod \"2dae8405-9ef5-4010-a5c6-a249869747e2\" (UID: \"2dae8405-9ef5-4010-a5c6-a249869747e2\") " Sep 10 03:04:16.147768 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147739 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2dae8405-9ef5-4010-a5c6-a249869747e2" (UID: "2dae8405-9ef5-4010-a5c6-a249869747e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:04:16.147993 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.147972 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dae8405-9ef5-4010-a5c6-a249869747e2-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:16.150507 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.150470 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2dae8405-9ef5-4010-a5c6-a249869747e2" (UID: "2dae8405-9ef5-4010-a5c6-a249869747e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:04:16.150617 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.150501 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph" (OuterVolumeSpecName: "kube-api-access-lvrph") pod "2dae8405-9ef5-4010-a5c6-a249869747e2" (UID: "2dae8405-9ef5-4010-a5c6-a249869747e2"). InnerVolumeSpecName "kube-api-access-lvrph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:04:16.150910 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.150882 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2dae8405-9ef5-4010-a5c6-a249869747e2" (UID: "2dae8405-9ef5-4010-a5c6-a249869747e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:04:16.248649 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.248611 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrph\" (UniqueName: \"kubernetes.io/projected/2dae8405-9ef5-4010-a5c6-a249869747e2-kube-api-access-lvrph\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:16.248649 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.248643 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:16.248649 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.248656 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dae8405-9ef5-4010-a5c6-a249869747e2-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:04:16.749666 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.749626 2583 generic.go:356] "Generic (PLEG): container finished" podID="2dae8405-9ef5-4010-a5c6-a249869747e2" containerID="42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d" exitCode=0 Sep 10 03:04:16.749904 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.749680 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" Sep 10 03:04:16.749904 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.749724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" event={"ID":"2dae8405-9ef5-4010-a5c6-a249869747e2","Type":"ContainerDied","Data":"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d"} Sep 10 03:04:16.749904 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.749791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6" event={"ID":"2dae8405-9ef5-4010-a5c6-a249869747e2","Type":"ContainerDied","Data":"2a851d58248197bf6b3b81d488af353f819a3683c9f693a41406de787b2c3b09"} Sep 10 03:04:16.749904 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.749825 2583 scope.go:117] "RemoveContainer" containerID="42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d" Sep 10 03:04:16.759618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.759598 2583 scope.go:117] "RemoveContainer" containerID="42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d" Sep 10 03:04:16.759923 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:04:16.759900 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d\": container with ID starting with 42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d not found: ID does not exist" containerID="42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d" Sep 10 03:04:16.760026 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.759930 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d"} err="failed to get container status \"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d\": rpc error: code = NotFound desc = could not find container \"42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d\": container with ID starting with 42d127d5d4fae64a5866c70e6b2b26d14692286b8b201bc08b49111b7160750d not found: ID does not exist" Sep 10 03:04:16.770198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.770173 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:04:16.775570 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:16.775549 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-af329f-5575b9786b-lplz6"] Sep 10 03:04:17.000870 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:17.000826 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dae8405-9ef5-4010-a5c6-a249869747e2" path="/var/lib/kubelet/pods/2dae8405-9ef5-4010-a5c6-a249869747e2/volumes" Sep 10 03:04:20.355046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355002 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:04:20.355567 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355548 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.355619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355570 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.355619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355580 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" containerName="maas-api" Sep 10 03:04:20.355619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355586 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" containerName="maas-api" Sep 10 03:04:20.355619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355603 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.355619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355609 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355626 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dae8405-9ef5-4010-a5c6-a249869747e2" containerName="maas-api" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355635 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dae8405-9ef5-4010-a5c6-a249869747e2" containerName="maas-api" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355698 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b2d053-e60c-4aae-91b4-61dfa0325c29" containerName="maas-api" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355713 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dae8405-9ef5-4010-a5c6-a249869747e2" containerName="maas-api" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355719 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.355792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.355726 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="026bb835-58e5-4017-a161-6a849913f261" containerName="istio-proxy" Sep 10 03:04:20.360259 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.360237 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.362810 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.362785 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1e0789-openshift-default-dockercfg-bdh7q\"" Sep 10 03:04:20.372446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.372416 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:04:20.488361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvnl\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488809 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.488809 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.488665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590043 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.589995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590043 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590306 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590306 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvnl\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590306 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590494 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590474 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590611 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590681 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590750 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590811 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.590811 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.590794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.591210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.591166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.591381 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.591362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.591524 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.591465 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.593108 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.593083 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.594211 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.594185 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.603876 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.603824 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.604029 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.603976 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvnl\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl\") pod \"e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.672477 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.672371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:20.951496 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.951466 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:04:20.953450 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:04:20.953424 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b51ce6_59b5_4604_bbb4_68ab88bb3471.slice/crio-f92b02ba6e12755140aa3c41450a999bbbb89e8aa8151ba0b2714041c7cc395c WatchSource:0}: Error finding container f92b02ba6e12755140aa3c41450a999bbbb89e8aa8151ba0b2714041c7cc395c: Status 404 returned error can't find the container with id f92b02ba6e12755140aa3c41450a999bbbb89e8aa8151ba0b2714041c7cc395c Sep 10 03:04:20.956241 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.956208 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:20.956322 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.956275 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:20.956322 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:20.956304 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:21.779194 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:21.779158 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerStarted","Data":"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27"} Sep 10 03:04:21.779680 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:21.779211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerStarted","Data":"f92b02ba6e12755140aa3c41450a999bbbb89e8aa8151ba0b2714041c7cc395c"} Sep 10 03:04:21.797553 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:21.797507 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podStartSLOduration=1.797491613 podStartE2EDuration="1.797491613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:21.796145971 +0000 UTC m=+1065.320002678" watchObservedRunningTime="2026-09-10 03:04:21.797491613 +0000 UTC m=+1065.321348296" Sep 10 03:04:22.673135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:22.673089 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:22.679074 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:22.679045 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:22.784774 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:22.784741 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:22.785705 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:22.785684 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:24.895262 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:24.895224 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:04:24.899094 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:24.899077 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:24.901338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:24.901308 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-1e0789-serving-cert\"" Sep 10 03:04:24.909362 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:24.909298 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:04:25.043160 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.043122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.043355 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.043229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnm5\" (UniqueName: \"kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.043355 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.043339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.043504 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.043482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.144603 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.144558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.144770 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.144623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnm5\" (UniqueName: \"kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.144852 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.144807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.145002 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.144982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.145777 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.145717 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.147389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.147364 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.147497 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.147419 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.152775 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.152746 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnm5\" (UniqueName: \"kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5\") pod \"maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.214310 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.214270 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:25.495756 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.495572 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:04:25.497838 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:04:25.497812 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda877c31a_d5a6_4317_88a8_e755893cc8eb.slice/crio-bb61121fa1a2a5fb16dc88e8d1a43fe76ae656e28b2888d0ab9bef989ba7eb95 WatchSource:0}: Error finding container bb61121fa1a2a5fb16dc88e8d1a43fe76ae656e28b2888d0ab9bef989ba7eb95: Status 404 returned error can't find the container with id bb61121fa1a2a5fb16dc88e8d1a43fe76ae656e28b2888d0ab9bef989ba7eb95 Sep 10 03:04:25.802667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:25.802625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" event={"ID":"a877c31a-d5a6-4317-88a8-e755893cc8eb","Type":"ContainerStarted","Data":"bb61121fa1a2a5fb16dc88e8d1a43fe76ae656e28b2888d0ab9bef989ba7eb95"} Sep 10 03:04:26.810179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:26.810141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" event={"ID":"a877c31a-d5a6-4317-88a8-e755893cc8eb","Type":"ContainerStarted","Data":"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426"} Sep 10 03:04:26.810567 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:26.810260 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:26.828812 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:26.828760 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" podStartSLOduration=2.354181554 podStartE2EDuration="2.828743183s" totalImagesPullingTime="474.561629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:26.826194957 +0000 UTC m=+1070.350051647" watchObservedRunningTime="2026-09-10 03:04:26.828743183 +0000 UTC m=+1070.352599870" Sep 10 03:04:32.821684 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:32.821606 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:04:49.481120 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.481083 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:04:49.487190 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.487163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.489567 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.489543 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c3058d-openshift-default-dockercfg-5xgrj\"" Sep 10 03:04:49.495622 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.495587 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:04:49.603880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.603615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.603880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.603693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604142 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.603918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604142 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604142 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604142 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604368 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhsp\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604368 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604215 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.604368 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.604243 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705423 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705612 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705460 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705666 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705800 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705838 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhsp\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.705990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.705981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.706210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.706023 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.706210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.706174 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.706590 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.706561 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.708542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.708508 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.708689 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.708672 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.713467 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.713439 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.713631 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.713528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhsp\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp\") pod \"e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:49.803004 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:49.802966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:50.082978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.082946 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:04:50.099053 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:04:50.099022 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5895f088_2562_4117_b5ed_65dd2c658f2f.slice/crio-c5a1334f6567dc2e45eb2d6b0f5106c0eef75d2a2325f35145a8abdad55412eb WatchSource:0}: Error finding container c5a1334f6567dc2e45eb2d6b0f5106c0eef75d2a2325f35145a8abdad55412eb: Status 404 returned error can't find the container with id c5a1334f6567dc2e45eb2d6b0f5106c0eef75d2a2325f35145a8abdad55412eb Sep 10 03:04:50.101477 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.101436 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.101569 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.101514 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.101569 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.101549 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.959654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.959615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerStarted","Data":"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7"} Sep 10 03:04:50.960182 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.959677 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerStarted","Data":"c5a1334f6567dc2e45eb2d6b0f5106c0eef75d2a2325f35145a8abdad55412eb"} Sep 10 03:04:50.961253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.961225 2583 generic.go:356] "Generic (PLEG): container finished" podID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerID="9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27" exitCode=0 Sep 10 03:04:50.961370 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.961271 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerDied","Data":"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27"} Sep 10 03:04:50.961648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.961634 2583 scope.go:117] "RemoveContainer" containerID="9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27" Sep 10 03:04:50.962758 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.962727 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.962834 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.962801 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.962887 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.962834 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:04:50.981191 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:50.981146 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podStartSLOduration=1.981131582 podStartE2EDuration="1.981131582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:50.978791861 +0000 UTC m=+1094.502648584" watchObservedRunningTime="2026-09-10 03:04:50.981131582 +0000 UTC m=+1094.504988268" Sep 10 03:04:51.804147 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:51.804088 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:51.810954 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:51.810913 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:51.969516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:51.969478 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerStarted","Data":"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088"} Sep 10 03:04:51.969937 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:51.969689 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:51.970596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:51.970578 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:04:52.673411 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:52.673369 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:52.674683 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:52.674656 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 10 03:04:52.674792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:52.674713 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 10 03:04:52.784958 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:52.784920 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:53.673675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:53.673637 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 10 03:04:53.674104 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:53.673716 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 10 03:04:54.650883 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.650835 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:04:54.654755 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.654724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.657108 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.656892 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c3058d-serving-cert\"" Sep 10 03:04:54.665242 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.665211 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:04:54.673011 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.672974 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 10 03:04:54.673163 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.673052 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 10 03:04:54.759795 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.759743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.760291 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.759831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nq5\" (UniqueName: \"kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.760291 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.759899 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.760291 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.759933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.860908 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.860875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nq5\" (UniqueName: \"kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.861091 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.860937 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.861091 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.860967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.861179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.861097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.861787 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.861757 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.863689 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.863662 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.863785 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.863748 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.870841 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.870814 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nq5\" (UniqueName: \"kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5\") pod \"maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:54.968109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:54.968016 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:55.303159 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:55.303121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:04:55.673845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:55.673753 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 10 03:04:55.673845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:55.673824 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 10 03:04:56.016053 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.016002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" event={"ID":"323ace4e-71d6-4dd0-9a94-72bb038d90e9","Type":"ContainerStarted","Data":"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0"} Sep 10 03:04:56.016492 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.016082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" event={"ID":"323ace4e-71d6-4dd0-9a94-72bb038d90e9","Type":"ContainerStarted","Data":"b091a0e9417a06f2c9404193cf0a11a94f72b0a4805d18df8b348d1a57da0af4"} Sep 10 03:04:56.016492 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.016271 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:04:56.037701 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.037637 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" podStartSLOduration=1.550474954 podStartE2EDuration="2.037615012s" totalImagesPullingTime="487.140058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:56.034758716 +0000 UTC m=+1099.558615431" watchObservedRunningTime="2026-09-10 03:04:56.037615012 +0000 UTC m=+1099.561471700" Sep 10 03:04:56.672964 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.672929 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 10 03:04:56.673225 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:56.672988 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 10 03:04:57.676982 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:57.676946 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:57.677956 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:57.677934 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:04:58.477431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:58.477398 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:04:58.477670 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:04:58.477645 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" containerID="cri-o://9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244" gracePeriod=30 Sep 10 03:05:02.028797 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:02.028765 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:05:03.242577 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.242537 2583 patch_prober.go:28] interesting pod/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 03:05:03.243022 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.242614 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 03:05:03.743022 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.742995 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:05:03.875746 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.875703 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.875994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.875762 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.875994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.875794 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.875994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.875921 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.875994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.875962 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.876238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876009 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.876238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876033 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.876238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876058 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdph6\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.876238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876090 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket\") pod \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\" (UID: \"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22\") " Sep 10 03:05:03.876238 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876171 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:05:03.876484 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876242 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:05:03.876484 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876421 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:05:03.876484 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876450 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:05:03.876648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876487 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:05:03.876648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876591 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.876648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876615 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.876648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876631 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.876648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876647 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.876842 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.876660 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.878041 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.878014 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:05:03.878201 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.878184 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:05:03.878594 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.878571 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6" (OuterVolumeSpecName: "kube-api-access-xdph6") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "kube-api-access-xdph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:05:03.878693 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.878597 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" (UID: "dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:05:03.977449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.977405 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.977449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.977439 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.977449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.977449 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdph6\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-kube-api-access-xdph6\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:03.977449 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:03.977460 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:05:04.064710 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.064676 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerID="9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244" exitCode=0 Sep 10 03:05:04.064914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.064723 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" Sep 10 03:05:04.064914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.064762 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerDied","Data":"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244"} Sep 10 03:05:04.064914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.064821 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq" event={"ID":"dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22","Type":"ContainerDied","Data":"660a49fb2a4a973eb6ba12cfaa71327dbc2c2f7450db42e88a5a53ef6447fec4"} Sep 10 03:05:04.064914 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.064845 2583 scope.go:117] "RemoveContainer" containerID="9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244" Sep 10 03:05:04.074486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.074464 2583 scope.go:117] "RemoveContainer" containerID="d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69" Sep 10 03:05:04.083721 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.083580 2583 scope.go:117] "RemoveContainer" containerID="9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244" Sep 10 03:05:04.084068 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:05:04.084040 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244\": container with ID starting with 9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244 not found: ID does not exist" containerID="9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244" Sep 10 03:05:04.084132 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.084084 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244"} err="failed to get container status \"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244\": rpc error: code = NotFound desc = could not find container \"9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244\": container with ID starting with 9e0ec08e772f1f98023663b67563730ba3610e7a59fbc6dfa1110f0724cca244 not found: ID does not exist" Sep 10 03:05:04.084132 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.084109 2583 scope.go:117] "RemoveContainer" containerID="d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69" Sep 10 03:05:04.084438 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:05:04.084417 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69\": container with ID starting with d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69 not found: ID does not exist" containerID="d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69" Sep 10 03:05:04.084512 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.084451 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69"} err="failed to get container status \"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69\": rpc error: code = NotFound desc = could not find container \"d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69\": container with ID starting with d3852f0422c5e2ea8ada32dc34f93aca9d0c906f6b4fc850d2efa2b787a3cd69 not found: ID does not exist" Sep 10 03:05:04.086282 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.086258 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:05:04.088788 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:04.088770 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-af329f-openshift-default-7f5d768f8d-4d7bq"] Sep 10 03:05:05.001495 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:05.001459 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" path="/var/lib/kubelet/pods/dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22/volumes" Sep 10 03:05:15.153293 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.153202 2583 generic.go:356] "Generic (PLEG): container finished" podID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerID="28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7" exitCode=0 Sep 10 03:05:15.153293 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.153272 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerDied","Data":"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7"} Sep 10 03:05:15.153821 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.153725 2583 scope.go:117] "RemoveContainer" containerID="28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7" Sep 10 03:05:15.154918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.154881 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:15.154989 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.154965 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:15.155029 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:15.154998 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:16.161373 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:16.161336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerStarted","Data":"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723"} Sep 10 03:05:16.803709 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:16.803664 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:05:16.803959 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:16.803804 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 03:05:16.803959 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:16.803844 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 03:05:17.803546 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:17.803496 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 03:05:17.804096 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:17.803572 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 03:05:18.807606 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:18.807569 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 03:05:18.808242 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:18.808208 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 03:05:19.804159 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:19.804108 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 03:05:19.804389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:19.804184 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 03:05:20.804121 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:20.804079 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 03:05:20.804773 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:20.804735 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 03:05:21.808416 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:21.808383 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:05:21.808843 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:21.808665 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:05:21.809474 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:21.809456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:05:28.524707 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.524667 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:05:28.525384 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.525363 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:28.525434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.525389 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:28.525535 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.525523 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:28.530303 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.530282 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.533037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.533014 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-61660fd5-openshift-default-dockercfg-trpn2\"" Sep 10 03:05:28.535991 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.535961 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:05:28.648269 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648471 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648471 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648471 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648421 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648739 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.648739 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.648700 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6hx\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750334 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750543 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750543 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750543 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750767 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6hx\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750767 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750767 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.750767 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751005 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.750802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751282 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.751232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751518 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.751467 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.751520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.751551 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.751737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.751647 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.753438 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.753410 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.754550 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.754499 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.758469 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.758442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6hx\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.758638 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.758622 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token\") pod \"e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:28.844684 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:28.844600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:29.120702 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.120670 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:05:29.122293 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:05:29.122259 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88073d7b_48f6_4f6c_a7e0_c061501e6425.slice/crio-7323761a58e598726d323034f4ea85ba8c39ed76450661dd4b351fdc8a2f39ce WatchSource:0}: Error finding container 7323761a58e598726d323034f4ea85ba8c39ed76450661dd4b351fdc8a2f39ce: Status 404 returned error can't find the container with id 7323761a58e598726d323034f4ea85ba8c39ed76450661dd4b351fdc8a2f39ce Sep 10 03:05:29.124912 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.124875 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:29.124993 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.124952 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:29.124993 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.124981 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:29.242117 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.242074 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerStarted","Data":"b7e000a08226b469ff05a418ed131671cf7be98c8ee7ac255ff7d233be51840e"} Sep 10 03:05:29.242298 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.242148 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerStarted","Data":"7323761a58e598726d323034f4ea85ba8c39ed76450661dd4b351fdc8a2f39ce"} Sep 10 03:05:29.259315 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.259256 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podStartSLOduration=1.259239436 podStartE2EDuration="1.259239436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:29.256821961 +0000 UTC m=+1132.780678649" watchObservedRunningTime="2026-09-10 03:05:29.259239436 +0000 UTC m=+1132.783096123" Sep 10 03:05:29.844875 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:29.844838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:30.850618 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:30.850583 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:31.254178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:31.254148 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:31.255233 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:31.255208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:32.331235 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.331194 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:05:32.332280 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.332251 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:32.332423 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.332397 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:32.332551 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.332537 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd88bcb-346a-4e6f-8ac2-a17ff3d92d22" containerName="istio-proxy" Sep 10 03:05:32.336115 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.336094 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.338036 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.338017 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61660fd5-serving-cert\"" Sep 10 03:05:32.346171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.346001 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:05:32.490285 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.490250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g776p\" (UniqueName: \"kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.490458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.490331 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.490458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.490368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.490458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.490431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.591389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.591299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g776p\" (UniqueName: \"kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.591389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.591384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.591606 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.591420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.591606 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.591458 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.592184 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.592162 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.593873 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.593830 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.593990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.593968 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.598671 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.598647 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g776p\" (UniqueName: \"kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p\") pod \"maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.651491 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.651459 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:32.928223 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:32.928194 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:05:32.930186 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:05:32.930151 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610d64ff_91a6_4669_929f_be14fd37734a.slice/crio-66b30573c2858480aba97d4d5f615c695442a94373404a0fc40348cb9bf17911 WatchSource:0}: Error finding container 66b30573c2858480aba97d4d5f615c695442a94373404a0fc40348cb9bf17911: Status 404 returned error can't find the container with id 66b30573c2858480aba97d4d5f615c695442a94373404a0fc40348cb9bf17911 Sep 10 03:05:33.267235 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:33.267194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" event={"ID":"610d64ff-91a6-4669-929f-be14fd37734a","Type":"ContainerStarted","Data":"66b30573c2858480aba97d4d5f615c695442a94373404a0fc40348cb9bf17911"} Sep 10 03:05:34.274770 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:34.274733 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" event={"ID":"610d64ff-91a6-4669-929f-be14fd37734a","Type":"ContainerStarted","Data":"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6"} Sep 10 03:05:34.275276 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:34.274819 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:34.290314 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:34.290251 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" podStartSLOduration=1.84252649 podStartE2EDuration="2.290229397s" totalImagesPullingTime="447.702907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:34.288779244 +0000 UTC m=+1137.812635929" watchObservedRunningTime="2026-09-10 03:05:34.290229397 +0000 UTC m=+1137.814086084" Sep 10 03:05:40.286493 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:40.286463 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:05:58.441480 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.441390 2583 generic.go:356] "Generic (PLEG): container finished" podID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerID="b7e000a08226b469ff05a418ed131671cf7be98c8ee7ac255ff7d233be51840e" exitCode=0 Sep 10 03:05:58.441480 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.441460 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerDied","Data":"b7e000a08226b469ff05a418ed131671cf7be98c8ee7ac255ff7d233be51840e"} Sep 10 03:05:58.442082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.441942 2583 scope.go:117] "RemoveContainer" containerID="b7e000a08226b469ff05a418ed131671cf7be98c8ee7ac255ff7d233be51840e" Sep 10 03:05:58.443314 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.443285 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:58.443394 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.443359 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:58.443432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:58.443392 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:05:59.450656 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:59.450566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerStarted","Data":"8420bc3d556582413a9585e1ba32b01210b98389f143caa8804938cc83a17c09"} Sep 10 03:05:59.845160 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:59.845126 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:05:59.846504 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:59.846475 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:05:59.846628 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:05:59.846529 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:00.845100 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:00.845059 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:06:00.845511 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:00.845125 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:01.254459 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:01.254426 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:06:01.845455 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:01.845416 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:06:01.845832 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:01.845491 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:02.450594 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.450545 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 03:06:02.450830 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.450797 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" podUID="6617bdd1-553e-4d50-95c3-693aafe448b3" containerName="maas-api" containerID="cri-o://ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61" gracePeriod=30 Sep 10 03:06:02.762329 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.762126 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 03:06:02.835880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.835678 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs\") pod \"6617bdd1-553e-4d50-95c3-693aafe448b3\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " Sep 10 03:06:02.835880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.835830 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") pod \"6617bdd1-553e-4d50-95c3-693aafe448b3\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " Sep 10 03:06:02.836133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.835927 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r826f\" (UniqueName: \"kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f\") pod \"6617bdd1-553e-4d50-95c3-693aafe448b3\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " Sep 10 03:06:02.836133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.835979 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca\") pod \"6617bdd1-553e-4d50-95c3-693aafe448b3\" (UID: \"6617bdd1-553e-4d50-95c3-693aafe448b3\") " Sep 10 03:06:02.837880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.836688 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6617bdd1-553e-4d50-95c3-693aafe448b3" (UID: "6617bdd1-553e-4d50-95c3-693aafe448b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:06:02.839590 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.839558 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6617bdd1-553e-4d50-95c3-693aafe448b3" (UID: "6617bdd1-553e-4d50-95c3-693aafe448b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:06:02.840263 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.840234 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6617bdd1-553e-4d50-95c3-693aafe448b3" (UID: "6617bdd1-553e-4d50-95c3-693aafe448b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:06:02.841183 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.840979 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f" (OuterVolumeSpecName: "kube-api-access-r826f") pod "6617bdd1-553e-4d50-95c3-693aafe448b3" (UID: "6617bdd1-553e-4d50-95c3-693aafe448b3"). InnerVolumeSpecName "kube-api-access-r826f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:06:02.845942 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.845912 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:06:02.846248 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.845972 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:02.937560 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.937523 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:02.937560 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.937552 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r826f\" (UniqueName: \"kubernetes.io/projected/6617bdd1-553e-4d50-95c3-693aafe448b3-kube-api-access-r826f\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:02.937560 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.937562 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617bdd1-553e-4d50-95c3-693aafe448b3-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:02.937560 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:02.937572 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617bdd1-553e-4d50-95c3-693aafe448b3-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:03.477264 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.477175 2583 generic.go:356] "Generic (PLEG): container finished" podID="6617bdd1-553e-4d50-95c3-693aafe448b3" containerID="ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61" exitCode=0 Sep 10 03:06:03.477264 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.477224 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" Sep 10 03:06:03.477447 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.477256 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" event={"ID":"6617bdd1-553e-4d50-95c3-693aafe448b3","Type":"ContainerDied","Data":"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61"} Sep 10 03:06:03.477447 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.477312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l" event={"ID":"6617bdd1-553e-4d50-95c3-693aafe448b3","Type":"ContainerDied","Data":"08dce3f87623e18e14d28be6b5e3b094692d12dff2d1cb73ffdbf20a73200167"} Sep 10 03:06:03.477447 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.477334 2583 scope.go:117] "RemoveContainer" containerID="ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61" Sep 10 03:06:03.494045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.488291 2583 scope.go:117] "RemoveContainer" containerID="ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61" Sep 10 03:06:03.494045 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:06:03.488683 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61\": container with ID starting with ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61 not found: ID does not exist" containerID="ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61" Sep 10 03:06:03.494045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.488721 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61"} err="failed to get container status \"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61\": rpc error: code = NotFound desc = could not find container \"ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61\": container with ID starting with ebd510840834f1fa26be6d7cc3c8caf4445dd4e1a6484de5a61646b4bcea3f61 not found: ID does not exist" Sep 10 03:06:03.499080 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.499048 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 03:06:03.511252 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.511208 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0f4da-dc4f4f47c-rpq7l"] Sep 10 03:06:03.845129 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.845088 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:06:03.845342 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:03.845151 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:04.845027 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:04.844986 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 03:06:04.845429 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:04.845064 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 03:06:05.006365 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:05.005951 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6617bdd1-553e-4d50-95c3-693aafe448b3" path="/var/lib/kubelet/pods/6617bdd1-553e-4d50-95c3-693aafe448b3/volumes" Sep 10 03:06:05.849733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:05.849702 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:06:05.850815 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:05.850794 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:06:08.943490 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:08.943456 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 03:06:08.943906 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:08.943714 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" containerID="cri-o://664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764" gracePeriod=30 Sep 10 03:06:14.209163 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.209099 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 03:06:14.359689 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359649 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.359903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359708 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.359903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359759 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.359903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359781 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.359903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359823 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt7j5\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.360133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.359925 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.360133 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360071 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.360241 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360139 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.360241 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360176 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data\") pod \"871779b9-64fe-47d5-a732-d0bea5edd793\" (UID: \"871779b9-64fe-47d5-a732-d0bea5edd793\") " Sep 10 03:06:14.360338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360245 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:06:14.360338 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360258 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:06:14.360671 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360620 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.360671 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360646 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.360826 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360689 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:06:14.360826 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360697 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:06:14.360826 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.360781 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data" (OuterVolumeSpecName: "istio-data") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:06:14.362847 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.362690 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:06:14.362847 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.362817 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:06:14.363229 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.363204 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5" (OuterVolumeSpecName: "kube-api-access-vt7j5") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "kube-api-access-vt7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:06:14.364537 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.364502 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token" (OuterVolumeSpecName: "istio-token") pod "871779b9-64fe-47d5-a732-d0bea5edd793" (UID: "871779b9-64fe-47d5-a732-d0bea5edd793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462327 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871779b9-64fe-47d5-a732-d0bea5edd793-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462377 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462389 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462400 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871779b9-64fe-47d5-a732-d0bea5edd793-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462411 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462423 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7j5\" (UniqueName: \"kubernetes.io/projected/871779b9-64fe-47d5-a732-d0bea5edd793-kube-api-access-vt7j5\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.462443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.462435 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871779b9-64fe-47d5-a732-d0bea5edd793-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:14.544996 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.544955 2583 generic.go:356] "Generic (PLEG): container finished" podID="871779b9-64fe-47d5-a732-d0bea5edd793" containerID="664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764" exitCode=0 Sep 10 03:06:14.545179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.545015 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" Sep 10 03:06:14.545179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.545044 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerDied","Data":"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764"} Sep 10 03:06:14.545179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.545102 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n" event={"ID":"871779b9-64fe-47d5-a732-d0bea5edd793","Type":"ContainerDied","Data":"72d7b8f94a18dbda7bb7590d9af37081e785a74336e98c346b98dc8472fb51dc"} Sep 10 03:06:14.545179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.545130 2583 scope.go:117] "RemoveContainer" containerID="664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764" Sep 10 03:06:14.557576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.557557 2583 scope.go:117] "RemoveContainer" containerID="072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14" Sep 10 03:06:14.567755 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.567725 2583 scope.go:117] "RemoveContainer" containerID="664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764" Sep 10 03:06:14.568108 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:06:14.568087 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764\": container with ID starting with 664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764 not found: ID does not exist" containerID="664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764" Sep 10 03:06:14.568169 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.568118 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764"} err="failed to get container status \"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764\": rpc error: code = NotFound desc = could not find container \"664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764\": container with ID starting with 664c43bcabf512e6f4c8820f98d2f143ca550220d050e55b6c8b4d0abf1ed764 not found: ID does not exist" Sep 10 03:06:14.568169 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.568137 2583 scope.go:117] "RemoveContainer" containerID="072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14" Sep 10 03:06:14.568442 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:06:14.568427 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14\": container with ID starting with 072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14 not found: ID does not exist" containerID="072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14" Sep 10 03:06:14.568479 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.568454 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 03:06:14.568522 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.568451 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14"} err="failed to get container status \"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14\": rpc error: code = NotFound desc = could not find container \"072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14\": container with ID starting with 072a0a3a2c490f845a7363a27dc5c48fe88c12bceffc9ba3c51797489a0efb14 not found: ID does not exist" Sep 10 03:06:14.572648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:14.572622 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b0f4da-openshift-default-775fb84548-q6s7n"] Sep 10 03:06:15.006161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:15.006124 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" path="/var/lib/kubelet/pods/871779b9-64fe-47d5-a732-d0bea5edd793/volumes" Sep 10 03:06:19.772532 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:19.772492 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 03:06:19.773034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:19.772698 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" podUID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" containerName="maas-api" containerID="cri-o://88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a" gracePeriod=30 Sep 10 03:06:20.044755 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.044401 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 03:06:20.120675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.120638 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk854\" (UniqueName: \"kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854\") pod \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " Sep 10 03:06:20.120833 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.120710 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs\") pod \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " Sep 10 03:06:20.120833 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.120773 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") pod \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " Sep 10 03:06:20.120833 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.120799 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca\") pod \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\" (UID: \"0c8404c5-c81c-4854-b86b-90de30fd5fe1\") " Sep 10 03:06:20.121223 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.121192 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c8404c5-c81c-4854-b86b-90de30fd5fe1" (UID: "0c8404c5-c81c-4854-b86b-90de30fd5fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:06:20.122952 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.122918 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854" (OuterVolumeSpecName: "kube-api-access-tk854") pod "0c8404c5-c81c-4854-b86b-90de30fd5fe1" (UID: "0c8404c5-c81c-4854-b86b-90de30fd5fe1"). InnerVolumeSpecName "kube-api-access-tk854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:06:20.122952 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.122924 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c8404c5-c81c-4854-b86b-90de30fd5fe1" (UID: "0c8404c5-c81c-4854-b86b-90de30fd5fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:06:20.123089 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.122983 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c8404c5-c81c-4854-b86b-90de30fd5fe1" (UID: "0c8404c5-c81c-4854-b86b-90de30fd5fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:06:20.222109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.222078 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:20.222109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.222105 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8404c5-c81c-4854-b86b-90de30fd5fe1-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:20.222109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.222114 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8404c5-c81c-4854-b86b-90de30fd5fe1-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:20.222109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.222123 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk854\" (UniqueName: \"kubernetes.io/projected/0c8404c5-c81c-4854-b86b-90de30fd5fe1-kube-api-access-tk854\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:06:20.580100 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.580057 2583 generic.go:356] "Generic (PLEG): container finished" podID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" containerID="88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a" exitCode=0 Sep 10 03:06:20.580302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.580120 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" Sep 10 03:06:20.580302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.580146 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" event={"ID":"0c8404c5-c81c-4854-b86b-90de30fd5fe1","Type":"ContainerDied","Data":"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a"} Sep 10 03:06:20.580302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.580234 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm" event={"ID":"0c8404c5-c81c-4854-b86b-90de30fd5fe1","Type":"ContainerDied","Data":"f06f3860f03eef0ff4a41d634746e57221d5647f863086f0f33bcfdc007c1d14"} Sep 10 03:06:20.580302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.580260 2583 scope.go:117] "RemoveContainer" containerID="88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a" Sep 10 03:06:20.590276 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.590257 2583 scope.go:117] "RemoveContainer" containerID="88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a" Sep 10 03:06:20.590593 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:06:20.590576 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a\": container with ID starting with 88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a not found: ID does not exist" containerID="88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a" Sep 10 03:06:20.590645 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.590600 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a"} err="failed to get container status \"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a\": rpc error: code = NotFound desc = could not find container \"88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a\": container with ID starting with 88915e1cccdb306c8cc48d782a513b146c0023e95016d4c7d1bbd2b42290525a not found: ID does not exist" Sep 10 03:06:20.602591 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.602561 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 03:06:20.606290 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:20.606259 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce5f9d-76d9b895c9-dspmm"] Sep 10 03:06:21.000827 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:21.000793 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" path="/var/lib/kubelet/pods/0c8404c5-c81c-4854-b86b-90de30fd5fe1/volumes" Sep 10 03:06:37.040291 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:37.040256 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:06:37.044149 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:06:37.044125 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:07:03.509378 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.509339 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:07:03.510116 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510094 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:03.510116 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510118 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:03.510268 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510128 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6617bdd1-553e-4d50-95c3-693aafe448b3" containerName="maas-api" Sep 10 03:07:03.510268 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510135 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6617bdd1-553e-4d50-95c3-693aafe448b3" containerName="maas-api" Sep 10 03:07:03.510268 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510168 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" containerName="maas-api" Sep 10 03:07:03.510268 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510176 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" containerName="maas-api" Sep 10 03:07:03.510465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510312 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8404c5-c81c-4854-b86b-90de30fd5fe1" containerName="maas-api" Sep 10 03:07:03.510465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510329 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:03.510465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510341 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6617bdd1-553e-4d50-95c3-693aafe448b3" containerName="maas-api" Sep 10 03:07:03.510465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.510353 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:03.515045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.515018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.517294 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.517270 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-af7727-openshift-default-dockercfg-h8hfz\"" Sep 10 03:07:03.525489 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.525454 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:07:03.570841 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.570796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.570845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.570911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.570994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.571038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.571110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.571156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.571221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.571267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.571250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.671842 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.671801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.671842 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.671848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.671918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.671951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.671993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672539 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672327 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672539 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672539 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672513 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672718 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672691 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.672824 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.672799 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.674475 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.674453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.674608 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.674579 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.679386 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.679357 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.679646 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.679628 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token\") pod \"e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:03.829984 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:03.829892 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:04.108003 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.107978 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:07:04.109936 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:07:04.109909 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7136d1_f59e_4948_b707_559b0a10b37b.slice/crio-d8875ec1869ca79203063e98b71ad723f75105d340926718c876d619774e24ab WatchSource:0}: Error finding container d8875ec1869ca79203063e98b71ad723f75105d340926718c876d619774e24ab: Status 404 returned error can't find the container with id d8875ec1869ca79203063e98b71ad723f75105d340926718c876d619774e24ab Sep 10 03:07:04.112484 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.112451 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:04.112572 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.112558 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:04.112612 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.112591 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:04.835650 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.835608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerStarted","Data":"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c"} Sep 10 03:07:04.836156 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.835708 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerStarted","Data":"d8875ec1869ca79203063e98b71ad723f75105d340926718c876d619774e24ab"} Sep 10 03:07:04.857851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:04.857792 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podStartSLOduration=1.8577761210000001 podStartE2EDuration="1.857776121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:04.85535714 +0000 UTC m=+1228.379213827" watchObservedRunningTime="2026-09-10 03:07:04.857776121 +0000 UTC m=+1228.381632867" Sep 10 03:07:05.830721 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:05.830676 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:05.836065 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:05.836035 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:05.840931 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:05.840906 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:05.841654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:05.841636 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:07.941301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.941271 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:07:07.941792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.941759 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:07.941792 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.941786 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="871779b9-64fe-47d5-a732-d0bea5edd793" containerName="istio-proxy" Sep 10 03:07:07.945194 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.945175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:07.947201 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.947179 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-af7727-serving-cert\"" Sep 10 03:07:07.955805 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:07.955543 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:07:08.017494 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.017456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.017684 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.017562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.017684 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.017591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lfs\" (UniqueName: \"kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.017684 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.017673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.118903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.118835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.119127 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.119005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.119127 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.119040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lfs\" (UniqueName: \"kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.119127 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.119099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.119260 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:07:08.119226 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-serving-cert: secret "maas-api-e2e-shared-a-w3-af7727-serving-cert" not found Sep 10 03:07:08.119296 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:07:08.119289 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls podName:0597dcb9-158f-4799-b71f-15ac02ebfe57 nodeName:}" failed. No retries permitted until 2026-09-10 03:07:08.619274457 +0000 UTC m=+1232.143131126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls") pod "maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" (UID: "0597dcb9-158f-4799-b71f-15ac02ebfe57") : secret "maas-api-e2e-shared-a-w3-af7727-serving-cert" not found Sep 10 03:07:08.119625 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.119600 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.121413 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.121390 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.126965 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.126940 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lfs\" (UniqueName: \"kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.625037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.625000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.627552 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.627518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:08.859415 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:08.859370 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:09.157007 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:09.156934 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:07:09.159252 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:07:09.159211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0597dcb9_158f_4799_b71f_15ac02ebfe57.slice/crio-188165a7890ede9bd9e69298e0f3b7386ea5c682a1af1260923efd5a65eb1aa2 WatchSource:0}: Error finding container 188165a7890ede9bd9e69298e0f3b7386ea5c682a1af1260923efd5a65eb1aa2: Status 404 returned error can't find the container with id 188165a7890ede9bd9e69298e0f3b7386ea5c682a1af1260923efd5a65eb1aa2 Sep 10 03:07:09.868658 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:09.868623 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" event={"ID":"0597dcb9-158f-4799-b71f-15ac02ebfe57","Type":"ContainerStarted","Data":"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294"} Sep 10 03:07:09.868845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:09.868680 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" event={"ID":"0597dcb9-158f-4799-b71f-15ac02ebfe57","Type":"ContainerStarted","Data":"188165a7890ede9bd9e69298e0f3b7386ea5c682a1af1260923efd5a65eb1aa2"} Sep 10 03:07:09.868845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:09.868741 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:09.886110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:09.886063 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" podStartSLOduration=2.399863664 podStartE2EDuration="2.886047116s" totalImagesPullingTime="486.183452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:09.885243729 +0000 UTC m=+1233.409100415" watchObservedRunningTime="2026-09-10 03:07:09.886047116 +0000 UTC m=+1233.409903803" Sep 10 03:07:15.880644 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:15.880615 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:07:37.166516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.165311 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:07:37.169389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.169365 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.171849 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.171542 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7daa14-serving-cert\"" Sep 10 03:07:37.179990 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.179968 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:07:37.236096 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.236045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.236096 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.236110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55dks\" (UniqueName: \"kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.236390 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.236277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.236390 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.236348 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.337232 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.337197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.337436 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.337245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.337436 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.337282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.337436 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.337312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55dks\" (UniqueName: \"kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.337436 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:07:37.337401 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-serving-cert: secret "maas-api-e2e-shared-b-w3-7daa14-serving-cert" not found Sep 10 03:07:37.337665 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:07:37.337455 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls podName:858e762c-c89e-44b6-9175-c9e6d4867710 nodeName:}" failed. No retries permitted until 2026-09-10 03:07:37.83744104 +0000 UTC m=+1261.361297710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" (UID: "858e762c-c89e-44b6-9175-c9e6d4867710") : secret "maas-api-e2e-shared-b-w3-7daa14-serving-cert" not found Sep 10 03:07:37.337992 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.337969 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.339644 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.339624 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.353371 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.353340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55dks\" (UniqueName: \"kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.843395 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.843357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:37.846404 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:37.846376 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:38.090509 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:38.090462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:38.377942 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:38.377908 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:07:38.380793 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:07:38.380765 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858e762c_c89e_44b6_9175_c9e6d4867710.slice/crio-ee586ba6fa7057dbbe32b0d9637b70904c6f7a3c84ad99424f282ac8bc8d42a2 WatchSource:0}: Error finding container ee586ba6fa7057dbbe32b0d9637b70904c6f7a3c84ad99424f282ac8bc8d42a2: Status 404 returned error can't find the container with id ee586ba6fa7057dbbe32b0d9637b70904c6f7a3c84ad99424f282ac8bc8d42a2 Sep 10 03:07:38.382239 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:38.382221 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 03:07:39.047057 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:39.046972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" event={"ID":"858e762c-c89e-44b6-9175-c9e6d4867710","Type":"ContainerStarted","Data":"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d"} Sep 10 03:07:39.047057 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:39.047026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" event={"ID":"858e762c-c89e-44b6-9175-c9e6d4867710","Type":"ContainerStarted","Data":"ee586ba6fa7057dbbe32b0d9637b70904c6f7a3c84ad99424f282ac8bc8d42a2"} Sep 10 03:07:39.047324 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:39.047306 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:39.066356 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:39.066298 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" podStartSLOduration=1.652343388 podStartE2EDuration="2.066282489s" totalImagesPullingTime="413.939101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:39.063508241 +0000 UTC m=+1262.587364929" watchObservedRunningTime="2026-09-10 03:07:39.066282489 +0000 UTC m=+1262.590139179" Sep 10 03:07:43.072156 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.072080 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerID="f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c" exitCode=0 Sep 10 03:07:43.072561 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.072155 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerDied","Data":"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c"} Sep 10 03:07:43.072629 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.072614 2583 scope.go:117] "RemoveContainer" containerID="f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c" Sep 10 03:07:43.073812 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.073785 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:43.073884 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.073869 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:43.073930 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:43.073903 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:07:44.079636 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:44.079602 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerStarted","Data":"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903"} Sep 10 03:07:44.830911 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:44.830851 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:44.831143 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:44.831120 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:44.831195 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:44.831174 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:45.059250 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:45.059214 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:07:45.831035 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:45.830993 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:45.831488 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:45.831056 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:46.830529 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:46.830486 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:46.830700 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:46.830550 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:47.830788 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:47.830753 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:47.831246 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:47.830819 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:48.830752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:48.830710 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:48.830980 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:48.830771 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:49.830436 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:49.830395 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 03:07:49.830660 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:49.830472 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 03:07:50.835079 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:50.835047 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:50.835462 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:50.835436 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:07:50.836205 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:07:50.836186 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:09:36.279578 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.279539 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:09:36.280109 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.279778 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" podUID="610d64ff-91a6-4669-929f-be14fd37734a" containerName="maas-api" containerID="cri-o://9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6" gracePeriod=30 Sep 10 03:09:36.547268 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.547192 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:09:36.639626 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.639594 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls\") pod \"610d64ff-91a6-4669-929f-be14fd37734a\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " Sep 10 03:09:36.639801 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.639647 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g776p\" (UniqueName: \"kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p\") pod \"610d64ff-91a6-4669-929f-be14fd37734a\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " Sep 10 03:09:36.639801 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.639757 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs\") pod \"610d64ff-91a6-4669-929f-be14fd37734a\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " Sep 10 03:09:36.639801 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.639783 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca\") pod \"610d64ff-91a6-4669-929f-be14fd37734a\" (UID: \"610d64ff-91a6-4669-929f-be14fd37734a\") " Sep 10 03:09:36.640259 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.640232 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "610d64ff-91a6-4669-929f-be14fd37734a" (UID: "610d64ff-91a6-4669-929f-be14fd37734a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:09:36.641780 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.641753 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "610d64ff-91a6-4669-929f-be14fd37734a" (UID: "610d64ff-91a6-4669-929f-be14fd37734a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:09:36.641906 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.641799 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p" (OuterVolumeSpecName: "kube-api-access-g776p") pod "610d64ff-91a6-4669-929f-be14fd37734a" (UID: "610d64ff-91a6-4669-929f-be14fd37734a"). InnerVolumeSpecName "kube-api-access-g776p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:09:36.641906 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.641889 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "610d64ff-91a6-4669-929f-be14fd37734a" (UID: "610d64ff-91a6-4669-929f-be14fd37734a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:09:36.725252 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.725215 2583 generic.go:356] "Generic (PLEG): container finished" podID="610d64ff-91a6-4669-929f-be14fd37734a" containerID="9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6" exitCode=0 Sep 10 03:09:36.725434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.725284 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" Sep 10 03:09:36.725434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.725300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" event={"ID":"610d64ff-91a6-4669-929f-be14fd37734a","Type":"ContainerDied","Data":"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6"} Sep 10 03:09:36.725434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.725354 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl" event={"ID":"610d64ff-91a6-4669-929f-be14fd37734a","Type":"ContainerDied","Data":"66b30573c2858480aba97d4d5f615c695442a94373404a0fc40348cb9bf17911"} Sep 10 03:09:36.725434 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.725377 2583 scope.go:117] "RemoveContainer" containerID="9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6" Sep 10 03:09:36.735515 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.735491 2583 scope.go:117] "RemoveContainer" containerID="9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6" Sep 10 03:09:36.735836 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:09:36.735815 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6\": container with ID starting with 9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6 not found: ID does not exist" containerID="9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6" Sep 10 03:09:36.735940 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.735849 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6"} err="failed to get container status \"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6\": rpc error: code = NotFound desc = could not find container \"9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6\": container with ID starting with 9213c1d5ba101c3b66fec4e2c7f2390956b460cf55180ea8c2eb14a3236affd6 not found: ID does not exist" Sep 10 03:09:36.740940 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.740915 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:09:36.741051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.740943 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610d64ff-91a6-4669-929f-be14fd37734a-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:09:36.741051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.740957 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610d64ff-91a6-4669-929f-be14fd37734a-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:09:36.741051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.740972 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g776p\" (UniqueName: \"kubernetes.io/projected/610d64ff-91a6-4669-929f-be14fd37734a-kube-api-access-g776p\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:09:36.747540 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.747504 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:09:36.751179 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:36.751146 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61660fd5-57c696dc7b-68gxl"] Sep 10 03:09:37.017216 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:37.017166 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610d64ff-91a6-4669-929f-be14fd37734a" path="/var/lib/kubelet/pods/610d64ff-91a6-4669-929f-be14fd37734a/volumes" Sep 10 03:09:59.413986 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.413951 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:09:59.414592 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.414494 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610d64ff-91a6-4669-929f-be14fd37734a" containerName="maas-api" Sep 10 03:09:59.414592 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.414512 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="610d64ff-91a6-4669-929f-be14fd37734a" containerName="maas-api" Sep 10 03:09:59.414716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.414648 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="610d64ff-91a6-4669-929f-be14fd37734a" containerName="maas-api" Sep 10 03:09:59.419495 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.419468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.423089 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.423065 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"kube-root-ca.crt\"" Sep 10 03:09:59.423236 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.423065 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"openshift-service-ca.crt\"" Sep 10 03:09:59.423236 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.423065 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"default-dockercfg-n5bs2\"" Sep 10 03:09:59.423422 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.423407 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"auth-test-model-7daa14-kserve-self-signed-certs\"" Sep 10 03:09:59.429354 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.429329 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591128 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whj7x\" (UniqueName: \"kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.592019 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.591397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.692894 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.692766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.692894 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.692809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.692894 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.692838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whj7x\" (UniqueName: \"kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693219 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.692922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693219 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.693059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693219 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.693118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693378 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.693321 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693378 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.693344 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.693672 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.693643 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.695301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.695274 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.695546 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.695529 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.701152 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.701126 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whj7x\" (UniqueName: \"kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x\") pod \"auth-test-model-7daa14-kserve-5555d945f-rcdch\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:09:59.730185 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:09:59.730147 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:10:00.005681 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:00.005651 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:10:00.007252 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:10:00.007218 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2892cbc8_c691_41b9_898e_a80eb19628ec.slice/crio-d600df0abc2ff1e9be7152a1dd1cce7c04be0a91a1df9c092aa3310295648d3e WatchSource:0}: Error finding container d600df0abc2ff1e9be7152a1dd1cce7c04be0a91a1df9c092aa3310295648d3e: Status 404 returned error can't find the container with id d600df0abc2ff1e9be7152a1dd1cce7c04be0a91a1df9c092aa3310295648d3e Sep 10 03:10:00.857905 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:00.857852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" event={"ID":"2892cbc8-c691-41b9-898e-a80eb19628ec","Type":"ContainerStarted","Data":"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7"} Sep 10 03:10:00.857905 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:00.857909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" event={"ID":"2892cbc8-c691-41b9-898e-a80eb19628ec","Type":"ContainerStarted","Data":"d600df0abc2ff1e9be7152a1dd1cce7c04be0a91a1df9c092aa3310295648d3e"} Sep 10 03:10:00.878753 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:00.878697 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" podStartSLOduration=1.878677573 podStartE2EDuration="1.878677573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:10:00.875393956 +0000 UTC m=+1404.399250656" watchObservedRunningTime="2026-09-10 03:10:00.878677573 +0000 UTC m=+1404.402534265" Sep 10 03:10:09.730289 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:09.730250 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:10:09.730789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:09.730310 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:10:09.743066 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:09.743036 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:10:09.929843 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:09.929812 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:10:33.936510 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:33.936413 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:10:33.937041 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:33.936674 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" containerID="cri-o://8420bc3d556582413a9585e1ba32b01210b98389f143caa8804938cc83a17c09" gracePeriod=30 Sep 10 03:10:35.852134 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:35.852091 2583 patch_prober.go:28] interesting pod/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 03:10:35.852532 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:35.852166 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 03:10:39.081900 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.081851 2583 generic.go:356] "Generic (PLEG): container finished" podID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerID="8420bc3d556582413a9585e1ba32b01210b98389f143caa8804938cc83a17c09" exitCode=0 Sep 10 03:10:39.082391 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.081903 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerDied","Data":"8420bc3d556582413a9585e1ba32b01210b98389f143caa8804938cc83a17c09"} Sep 10 03:10:39.082391 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.081978 2583 scope.go:117] "RemoveContainer" containerID="b7e000a08226b469ff05a418ed131671cf7be98c8ee7ac255ff7d233be51840e" Sep 10 03:10:39.195110 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.195074 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:10:39.223748 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223647 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.223748 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223709 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq6hx\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223752 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223876 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223936 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.223964 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data" (OuterVolumeSpecName: "istio-data") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224029 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224116 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224156 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224164 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224189 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224214 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token\") pod \"88073d7b-48f6-4f6c-a7e0-c061501e6425\" (UID: \"88073d7b-48f6-4f6c-a7e0-c061501e6425\") " Sep 10 03:10:39.224583 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224309 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:10:39.224583 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224517 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:10:39.224722 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224694 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.224782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224722 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.224782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224737 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.224782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224752 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.224782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.224766 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88073d7b-48f6-4f6c-a7e0-c061501e6425-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.226401 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.226375 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:10:39.226602 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.226577 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:10:39.226752 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.226736 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx" (OuterVolumeSpecName: "kube-api-access-gq6hx") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "kube-api-access-gq6hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:10:39.226824 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.226806 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token" (OuterVolumeSpecName: "istio-token") pod "88073d7b-48f6-4f6c-a7e0-c061501e6425" (UID: "88073d7b-48f6-4f6c-a7e0-c061501e6425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:10:39.325808 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.325768 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.325808 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.325803 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.325808 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.325812 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq6hx\" (UniqueName: \"kubernetes.io/projected/88073d7b-48f6-4f6c-a7e0-c061501e6425-kube-api-access-gq6hx\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:39.326083 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:39.325822 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88073d7b-48f6-4f6c-a7e0-c061501e6425-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:10:40.089615 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:40.089575 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" event={"ID":"88073d7b-48f6-4f6c-a7e0-c061501e6425","Type":"ContainerDied","Data":"7323761a58e598726d323034f4ea85ba8c39ed76450661dd4b351fdc8a2f39ce"} Sep 10 03:10:40.090114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:40.089647 2583 scope.go:117] "RemoveContainer" containerID="8420bc3d556582413a9585e1ba32b01210b98389f143caa8804938cc83a17c09" Sep 10 03:10:40.090114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:40.089588 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh" Sep 10 03:10:40.112577 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:40.112548 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:10:40.118231 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:40.118203 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-61660fd5-openshift-default-75cf76b6d8-jcfmh"] Sep 10 03:10:41.001787 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:10:41.001753 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" path="/var/lib/kubelet/pods/88073d7b-48f6-4f6c-a7e0-c061501e6425/volumes" Sep 10 03:11:27.745915 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.745848 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746353 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746367 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746387 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746392 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746467 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.746486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.746479 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88073d7b-48f6-4f6c-a7e0-c061501e6425" containerName="istio-proxy" Sep 10 03:11:27.750202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.750179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.752733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.752705 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f0a342-openshift-default-dockercfg-gfzkh\"" Sep 10 03:11:27.762597 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.762566 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:11:27.839842 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.839793 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840068 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.839949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840068 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddvrb\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840257 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840257 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840257 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.840257 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.840251 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941457 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941499 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddvrb\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.941654 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.941690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.942015 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942185 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.942166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942248 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.942193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.942420 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.942595 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.942508 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.944046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.944022 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.944332 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.944313 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.954974 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.954944 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:27.955210 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:27.955190 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddvrb\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb\") pod \"e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:28.063090 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.063050 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:28.342083 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.342025 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:11:28.344454 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:11:28.344427 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab26a95_d2c6_453d_a366_20f8e2a4d24d.slice/crio-37dfca74ed8c3d57fe2ee5679935dd9940ef984b6da02ba2b7ce5f553f6d4e69 WatchSource:0}: Error finding container 37dfca74ed8c3d57fe2ee5679935dd9940ef984b6da02ba2b7ce5f553f6d4e69: Status 404 returned error can't find the container with id 37dfca74ed8c3d57fe2ee5679935dd9940ef984b6da02ba2b7ce5f553f6d4e69 Sep 10 03:11:28.346720 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.346685 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:28.346803 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.346755 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:28.346803 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.346791 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:28.369069 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:28.369034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerStarted","Data":"37dfca74ed8c3d57fe2ee5679935dd9940ef984b6da02ba2b7ce5f553f6d4e69"} Sep 10 03:11:29.375909 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:29.375853 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerStarted","Data":"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582"} Sep 10 03:11:29.396746 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:29.396690 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podStartSLOduration=2.396672777 podStartE2EDuration="2.396672777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:11:29.394128299 +0000 UTC m=+1492.917984986" watchObservedRunningTime="2026-09-10 03:11:29.396672777 +0000 UTC m=+1492.920529463" Sep 10 03:11:30.063533 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:30.063499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:30.068899 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:30.068853 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:30.381547 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:30.381468 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:30.382551 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:30.382530 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:32.429874 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.429816 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:11:32.434050 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.434016 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.436215 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.436189 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f0a342-serving-cert\"" Sep 10 03:11:32.448823 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.448795 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:11:32.593153 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.593112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.593329 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.593238 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.593329 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.593277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sdk\" (UniqueName: \"kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.593410 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.593388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.694442 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.694343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.694634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.694455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.694634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.694500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sdk\" (UniqueName: \"kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.694634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.694581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.695146 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.695123 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.697039 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.697012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.697143 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.697070 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.702698 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.702673 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sdk\" (UniqueName: \"kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk\") pod \"maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:32.747629 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:32.747592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:33.023032 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.023002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:11:33.024556 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:11:33.024526 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe762343_05c9_4e7c_b4af_85b6c58071a2.slice/crio-afe90099b0f6b8e6c775c36ad28c17235e98ef6b52570e35ecbe5464222f483f WatchSource:0}: Error finding container afe90099b0f6b8e6c775c36ad28c17235e98ef6b52570e35ecbe5464222f483f: Status 404 returned error can't find the container with id afe90099b0f6b8e6c775c36ad28c17235e98ef6b52570e35ecbe5464222f483f Sep 10 03:11:33.236067 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.236031 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 03:11:33.236374 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.236310 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" podUID="60b5504d-a5ab-4648-b239-5ee95322b03d" containerName="maas-api" containerID="cri-o://cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827" gracePeriod=30 Sep 10 03:11:33.398537 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.398442 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" event={"ID":"be762343-05c9-4e7c-b4af-85b6c58071a2","Type":"ContainerStarted","Data":"afe90099b0f6b8e6c775c36ad28c17235e98ef6b52570e35ecbe5464222f483f"} Sep 10 03:11:33.680830 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.680805 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 03:11:33.808525 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.808475 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") pod \"60b5504d-a5ab-4648-b239-5ee95322b03d\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " Sep 10 03:11:33.808741 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.808568 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca\") pod \"60b5504d-a5ab-4648-b239-5ee95322b03d\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " Sep 10 03:11:33.808741 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.808674 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpvjm\" (UniqueName: \"kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm\") pod \"60b5504d-a5ab-4648-b239-5ee95322b03d\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " Sep 10 03:11:33.808909 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.808741 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs\") pod \"60b5504d-a5ab-4648-b239-5ee95322b03d\" (UID: \"60b5504d-a5ab-4648-b239-5ee95322b03d\") " Sep 10 03:11:33.808987 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.808964 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60b5504d-a5ab-4648-b239-5ee95322b03d" (UID: "60b5504d-a5ab-4648-b239-5ee95322b03d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:11:33.809270 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.809250 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b5504d-a5ab-4648-b239-5ee95322b03d-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:11:33.810892 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.810829 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60b5504d-a5ab-4648-b239-5ee95322b03d" (UID: "60b5504d-a5ab-4648-b239-5ee95322b03d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:11:33.811001 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.810892 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60b5504d-a5ab-4648-b239-5ee95322b03d" (UID: "60b5504d-a5ab-4648-b239-5ee95322b03d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:11:33.811103 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.811003 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm" (OuterVolumeSpecName: "kube-api-access-dpvjm") pod "60b5504d-a5ab-4648-b239-5ee95322b03d" (UID: "60b5504d-a5ab-4648-b239-5ee95322b03d"). InnerVolumeSpecName "kube-api-access-dpvjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:11:33.910851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.910812 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvjm\" (UniqueName: \"kubernetes.io/projected/60b5504d-a5ab-4648-b239-5ee95322b03d-kube-api-access-dpvjm\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:11:33.910851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.910843 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:11:33.910851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:33.910852 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b5504d-a5ab-4648-b239-5ee95322b03d-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:11:34.405212 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.405179 2583 generic.go:356] "Generic (PLEG): container finished" podID="60b5504d-a5ab-4648-b239-5ee95322b03d" containerID="cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827" exitCode=0 Sep 10 03:11:34.405415 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.405216 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" event={"ID":"60b5504d-a5ab-4648-b239-5ee95322b03d","Type":"ContainerDied","Data":"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827"} Sep 10 03:11:34.405415 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.405238 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" Sep 10 03:11:34.405415 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.405266 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s" event={"ID":"60b5504d-a5ab-4648-b239-5ee95322b03d","Type":"ContainerDied","Data":"ebc0a12b29f593cb7b47f933d6aa655ba19afdf180fd1c8c3525c6bb2d2f9c4f"} Sep 10 03:11:34.405415 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.405299 2583 scope.go:117] "RemoveContainer" containerID="cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827" Sep 10 03:11:34.407290 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.407264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" event={"ID":"be762343-05c9-4e7c-b4af-85b6c58071a2","Type":"ContainerStarted","Data":"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d"} Sep 10 03:11:34.407427 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.407332 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:34.415547 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.415528 2583 scope.go:117] "RemoveContainer" containerID="cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827" Sep 10 03:11:34.415965 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:11:34.415942 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827\": container with ID starting with cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827 not found: ID does not exist" containerID="cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827" Sep 10 03:11:34.416050 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.415981 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827"} err="failed to get container status \"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827\": rpc error: code = NotFound desc = could not find container \"cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827\": container with ID starting with cd0c161f6e608d8e144f5e9a79fc1c1d530f12e855ec46032ab729826d8eb827 not found: ID does not exist" Sep 10 03:11:34.455917 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.455876 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 03:11:34.482959 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.482922 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-097e16-b8cf4899f-6r72s"] Sep 10 03:11:34.513524 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:34.513470 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" podStartSLOduration=2.060787119 podStartE2EDuration="2.513452941s" totalImagesPullingTime="452.665822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:11:34.510911119 +0000 UTC m=+1498.034767807" watchObservedRunningTime="2026-09-10 03:11:34.513452941 +0000 UTC m=+1498.037309628" Sep 10 03:11:35.003031 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:35.002999 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b5504d-a5ab-4648-b239-5ee95322b03d" path="/var/lib/kubelet/pods/60b5504d-a5ab-4648-b239-5ee95322b03d/volumes" Sep 10 03:11:37.084845 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:37.083479 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:11:37.091998 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:37.091967 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:11:40.420073 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:40.420037 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:11:57.563685 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.563648 2583 generic.go:356] "Generic (PLEG): container finished" podID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerID="7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582" exitCode=0 Sep 10 03:11:57.564251 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.563721 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerDied","Data":"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582"} Sep 10 03:11:57.564320 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.564291 2583 scope.go:117] "RemoveContainer" containerID="7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582" Sep 10 03:11:57.565610 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.565574 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:57.565697 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.565656 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:57.565697 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:57.565689 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:11:58.571596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:58.571555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerStarted","Data":"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02"} Sep 10 03:11:59.064000 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:59.063907 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:11:59.065336 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:59.065297 2583 patch_prober.go:28] interesting pod/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 03:11:59.065466 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:11:59.065367 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 03:12:00.063853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:00.063814 2583 patch_prober.go:28] interesting pod/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 03:12:00.064267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:00.063896 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 03:12:00.382028 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:00.381942 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:12:01.063443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:01.063408 2583 patch_prober.go:28] interesting pod/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 03:12:01.063627 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:01.063459 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 03:12:02.063540 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:02.063503 2583 patch_prober.go:28] interesting pod/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 03:12:02.063977 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:02.063565 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 03:12:03.063462 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:03.063423 2583 patch_prober.go:28] interesting pod/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 10 03:12:03.063652 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:03.063502 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 10 03:12:04.067704 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:04.067669 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:12:04.068825 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:04.068802 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:12:44.025375 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.025340 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:12:44.025930 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.025570 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" podUID="be762343-05c9-4e7c-b4af-85b6c58071a2" containerName="maas-api" containerID="cri-o://f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d" gracePeriod=30 Sep 10 03:12:44.295940 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.295908 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:12:44.412034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.411993 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7sdk\" (UniqueName: \"kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk\") pod \"be762343-05c9-4e7c-b4af-85b6c58071a2\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " Sep 10 03:12:44.412234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.412058 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls\") pod \"be762343-05c9-4e7c-b4af-85b6c58071a2\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " Sep 10 03:12:44.412234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.412100 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca\") pod \"be762343-05c9-4e7c-b4af-85b6c58071a2\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " Sep 10 03:12:44.412234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.412217 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs\") pod \"be762343-05c9-4e7c-b4af-85b6c58071a2\" (UID: \"be762343-05c9-4e7c-b4af-85b6c58071a2\") " Sep 10 03:12:44.412580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.412537 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be762343-05c9-4e7c-b4af-85b6c58071a2" (UID: "be762343-05c9-4e7c-b4af-85b6c58071a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:12:44.412853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.412832 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be762343-05c9-4e7c-b4af-85b6c58071a2-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:44.414430 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.414400 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be762343-05c9-4e7c-b4af-85b6c58071a2" (UID: "be762343-05c9-4e7c-b4af-85b6c58071a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:12:44.414907 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.414883 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be762343-05c9-4e7c-b4af-85b6c58071a2" (UID: "be762343-05c9-4e7c-b4af-85b6c58071a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:12:44.414983 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.414905 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk" (OuterVolumeSpecName: "kube-api-access-n7sdk") pod "be762343-05c9-4e7c-b4af-85b6c58071a2" (UID: "be762343-05c9-4e7c-b4af-85b6c58071a2"). InnerVolumeSpecName "kube-api-access-n7sdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:12:44.513596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.513561 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:44.513596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.513593 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7sdk\" (UniqueName: \"kubernetes.io/projected/be762343-05c9-4e7c-b4af-85b6c58071a2-kube-api-access-n7sdk\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:44.513596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.513602 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be762343-05c9-4e7c-b4af-85b6c58071a2-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:44.833670 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.833637 2583 generic.go:356] "Generic (PLEG): container finished" podID="be762343-05c9-4e7c-b4af-85b6c58071a2" containerID="f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d" exitCode=0 Sep 10 03:12:44.833851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.833683 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" Sep 10 03:12:44.833851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.833729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" event={"ID":"be762343-05c9-4e7c-b4af-85b6c58071a2","Type":"ContainerDied","Data":"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d"} Sep 10 03:12:44.833851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.833792 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn" event={"ID":"be762343-05c9-4e7c-b4af-85b6c58071a2","Type":"ContainerDied","Data":"afe90099b0f6b8e6c775c36ad28c17235e98ef6b52570e35ecbe5464222f483f"} Sep 10 03:12:44.833851 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.833819 2583 scope.go:117] "RemoveContainer" containerID="f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d" Sep 10 03:12:44.844003 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.843980 2583 scope.go:117] "RemoveContainer" containerID="f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d" Sep 10 03:12:44.844313 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:12:44.844293 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d\": container with ID starting with f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d not found: ID does not exist" containerID="f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d" Sep 10 03:12:44.844367 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.844323 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d"} err="failed to get container status \"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d\": rpc error: code = NotFound desc = could not find container \"f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d\": container with ID starting with f4988427b65767bb0685d7d8ae099a0eb718937bd23ead121287043ed039f07d not found: ID does not exist" Sep 10 03:12:44.858468 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.858435 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:12:44.860650 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:44.860626 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0a342-6657bc5b7f-68qxn"] Sep 10 03:12:45.001619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:45.001584 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be762343-05c9-4e7c-b4af-85b6c58071a2" path="/var/lib/kubelet/pods/be762343-05c9-4e7c-b4af-85b6c58071a2/volumes" Sep 10 03:12:50.841875 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:50.841830 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:12:50.842293 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:50.842058 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" containerID="cri-o://b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02" gracePeriod=30 Sep 10 03:12:56.113361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.113279 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:12:56.241952 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.241907 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.241987 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242040 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242065 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242092 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242162 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242150 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242221 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:12:56.242431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242239 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242298 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242312 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:12:56.242431 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242343 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddvrb\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb\") pod \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\" (UID: \"cab26a95-d2c6-453d-a366-20f8e2a4d24d\") " Sep 10 03:12:56.242702 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242432 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:12:56.242702 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242547 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:12:56.242702 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242588 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:12:56.242950 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242748 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.242950 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242763 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.242950 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242774 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.242950 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242783 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.242950 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.242791 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.244429 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.244404 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:12:56.244563 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.244515 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb" (OuterVolumeSpecName: "kube-api-access-ddvrb") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "kube-api-access-ddvrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:12:56.244786 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.244759 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:12:56.244786 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.244763 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cab26a95-d2c6-453d-a366-20f8e2a4d24d" (UID: "cab26a95-d2c6-453d-a366-20f8e2a4d24d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:12:56.344198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.344157 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.344198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.344189 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.344198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.344198 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab26a95-d2c6-453d-a366-20f8e2a4d24d-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.344198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.344207 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddvrb\" (UniqueName: \"kubernetes.io/projected/cab26a95-d2c6-453d-a366-20f8e2a4d24d-kube-api-access-ddvrb\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:12:56.904167 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.904129 2583 generic.go:356] "Generic (PLEG): container finished" podID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerID="b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02" exitCode=0 Sep 10 03:12:56.904376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.904192 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" Sep 10 03:12:56.904376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.904211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerDied","Data":"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02"} Sep 10 03:12:56.904376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.904267 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj" event={"ID":"cab26a95-d2c6-453d-a366-20f8e2a4d24d","Type":"ContainerDied","Data":"37dfca74ed8c3d57fe2ee5679935dd9940ef984b6da02ba2b7ce5f553f6d4e69"} Sep 10 03:12:56.904376 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.904290 2583 scope.go:117] "RemoveContainer" containerID="b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02" Sep 10 03:12:56.914963 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.914945 2583 scope.go:117] "RemoveContainer" containerID="7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582" Sep 10 03:12:56.924580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.924558 2583 scope.go:117] "RemoveContainer" containerID="b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02" Sep 10 03:12:56.924939 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:12:56.924911 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02\": container with ID starting with b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02 not found: ID does not exist" containerID="b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02" Sep 10 03:12:56.925012 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.924948 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02"} err="failed to get container status \"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02\": rpc error: code = NotFound desc = could not find container \"b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02\": container with ID starting with b7581a22119dcdcb644f080ce82debfe9d514d5346da518eb61eac344fb28a02 not found: ID does not exist" Sep 10 03:12:56.925012 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.924967 2583 scope.go:117] "RemoveContainer" containerID="7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582" Sep 10 03:12:56.925286 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:12:56.925265 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582\": container with ID starting with 7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582 not found: ID does not exist" containerID="7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582" Sep 10 03:12:56.925352 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.925300 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582"} err="failed to get container status \"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582\": rpc error: code = NotFound desc = could not find container \"7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582\": container with ID starting with 7c74412e794f7adb2706cb05a40a04f1699757959912a85548d3e89520bed582 not found: ID does not exist" Sep 10 03:12:56.926790 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.926771 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:12:56.933184 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:56.933159 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f0a342-openshift-default-5d964f588d-dzqdj"] Sep 10 03:12:57.001083 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:12:57.001049 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" path="/var/lib/kubelet/pods/cab26a95-d2c6-453d-a366-20f8e2a4d24d/volumes" Sep 10 03:13:25.196678 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.196640 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:13:25.197223 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.196880 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" podUID="a877c31a-d5a6-4317-88a8-e755893cc8eb" containerName="maas-api" containerID="cri-o://bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426" gracePeriod=30 Sep 10 03:13:25.466544 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.466446 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:13:25.551396 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.551358 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs\") pod \"a877c31a-d5a6-4317-88a8-e755893cc8eb\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " Sep 10 03:13:25.551604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.551544 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rnm5\" (UniqueName: \"kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5\") pod \"a877c31a-d5a6-4317-88a8-e755893cc8eb\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " Sep 10 03:13:25.551677 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.551604 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca\") pod \"a877c31a-d5a6-4317-88a8-e755893cc8eb\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " Sep 10 03:13:25.551799 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.551776 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls\") pod \"a877c31a-d5a6-4317-88a8-e755893cc8eb\" (UID: \"a877c31a-d5a6-4317-88a8-e755893cc8eb\") " Sep 10 03:13:25.552197 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.552023 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a877c31a-d5a6-4317-88a8-e755893cc8eb" (UID: "a877c31a-d5a6-4317-88a8-e755893cc8eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:13:25.552312 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.552228 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a877c31a-d5a6-4317-88a8-e755893cc8eb-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:25.553969 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.553925 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5" (OuterVolumeSpecName: "kube-api-access-6rnm5") pod "a877c31a-d5a6-4317-88a8-e755893cc8eb" (UID: "a877c31a-d5a6-4317-88a8-e755893cc8eb"). InnerVolumeSpecName "kube-api-access-6rnm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:25.554191 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.554170 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a877c31a-d5a6-4317-88a8-e755893cc8eb" (UID: "a877c31a-d5a6-4317-88a8-e755893cc8eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:13:25.554371 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.554341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a877c31a-d5a6-4317-88a8-e755893cc8eb" (UID: "a877c31a-d5a6-4317-88a8-e755893cc8eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:13:25.652836 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.652787 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:25.652836 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.652833 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a877c31a-d5a6-4317-88a8-e755893cc8eb-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:25.652836 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:25.652844 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnm5\" (UniqueName: \"kubernetes.io/projected/a877c31a-d5a6-4317-88a8-e755893cc8eb-kube-api-access-6rnm5\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:26.073088 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.073053 2583 generic.go:356] "Generic (PLEG): container finished" podID="a877c31a-d5a6-4317-88a8-e755893cc8eb" containerID="bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426" exitCode=0 Sep 10 03:13:26.073271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.073101 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" Sep 10 03:13:26.073271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.073148 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" event={"ID":"a877c31a-d5a6-4317-88a8-e755893cc8eb","Type":"ContainerDied","Data":"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426"} Sep 10 03:13:26.073271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.073206 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt" event={"ID":"a877c31a-d5a6-4317-88a8-e755893cc8eb","Type":"ContainerDied","Data":"bb61121fa1a2a5fb16dc88e8d1a43fe76ae656e28b2888d0ab9bef989ba7eb95"} Sep 10 03:13:26.073271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.073228 2583 scope.go:117] "RemoveContainer" containerID="bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426" Sep 10 03:13:26.082748 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.082729 2583 scope.go:117] "RemoveContainer" containerID="bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426" Sep 10 03:13:26.083041 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:26.083020 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426\": container with ID starting with bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426 not found: ID does not exist" containerID="bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426" Sep 10 03:13:26.083126 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.083056 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426"} err="failed to get container status \"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426\": rpc error: code = NotFound desc = could not find container \"bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426\": container with ID starting with bc4a468b3de17e3feb6ef8c994bd91399cd24ae800b554f30b9892b94da9f426 not found: ID does not exist" Sep 10 03:13:26.104563 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.104531 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:13:26.118987 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:26.118953 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1e0789-7b4f65d8f4-9gtvt"] Sep 10 03:13:27.000708 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:27.000673 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a877c31a-d5a6-4317-88a8-e755893cc8eb" path="/var/lib/kubelet/pods/a877c31a-d5a6-4317-88a8-e755893cc8eb/volumes" Sep 10 03:13:32.250217 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:32.250133 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:13:32.250698 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:32.250382 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" containerID="cri-o://634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088" gracePeriod=30 Sep 10 03:13:37.510917 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.510889 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:13:37.682746 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682710 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.682955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682762 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbvnl\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.682955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682792 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.682955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682884 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682957 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.682991 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683046 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683114 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683081 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683327 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683140 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy\") pod \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\" (UID: \"e5b51ce6-59b5-4604-bbb4-68ab88bb3471\") " Sep 10 03:13:37.683327 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683283 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:37.683327 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683297 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:37.683327 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683312 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:37.683513 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683417 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:13:37.683617 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683595 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.683675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683623 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.683675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683620 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:37.683675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683638 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.683675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.683652 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.685423 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.685397 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:37.685423 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.685404 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:37.685604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.685564 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:13:37.685648 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.685599 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl" (OuterVolumeSpecName: "kube-api-access-lbvnl") pod "e5b51ce6-59b5-4604-bbb4-68ab88bb3471" (UID: "e5b51ce6-59b5-4604-bbb4-68ab88bb3471"). InnerVolumeSpecName "kube-api-access-lbvnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:37.784586 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.784548 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.784586 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.784582 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.784586 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.784591 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.784850 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.784601 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbvnl\" (UniqueName: \"kubernetes.io/projected/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-kube-api-access-lbvnl\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:37.784850 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:37.784610 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b51ce6-59b5-4604-bbb4-68ab88bb3471-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:38.149473 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.149378 2583 generic.go:356] "Generic (PLEG): container finished" podID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerID="634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088" exitCode=0 Sep 10 03:13:38.149473 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.149434 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" Sep 10 03:13:38.149473 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.149441 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerDied","Data":"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088"} Sep 10 03:13:38.149733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.149495 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v" event={"ID":"e5b51ce6-59b5-4604-bbb4-68ab88bb3471","Type":"ContainerDied","Data":"f92b02ba6e12755140aa3c41450a999bbbb89e8aa8151ba0b2714041c7cc395c"} Sep 10 03:13:38.149733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.149525 2583 scope.go:117] "RemoveContainer" containerID="634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088" Sep 10 03:13:38.159306 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.159289 2583 scope.go:117] "RemoveContainer" containerID="9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27" Sep 10 03:13:38.167425 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.167399 2583 scope.go:117] "RemoveContainer" containerID="634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088" Sep 10 03:13:38.167982 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:38.167952 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088\": container with ID starting with 634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088 not found: ID does not exist" containerID="634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088" Sep 10 03:13:38.168119 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.167994 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088"} err="failed to get container status \"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088\": rpc error: code = NotFound desc = could not find container \"634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088\": container with ID starting with 634e8f2fe03999fef79697998281835f04471b20568a4733d7e4ff4366c45088 not found: ID does not exist" Sep 10 03:13:38.168119 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.168019 2583 scope.go:117] "RemoveContainer" containerID="9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27" Sep 10 03:13:38.168433 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:38.168408 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27\": container with ID starting with 9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27 not found: ID does not exist" containerID="9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27" Sep 10 03:13:38.168516 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.168442 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27"} err="failed to get container status \"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27\": rpc error: code = NotFound desc = could not find container \"9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27\": container with ID starting with 9a1544b71f7f85f590f962558e17fa28b7df6a6490a283b077b76d645abf4b27 not found: ID does not exist" Sep 10 03:13:38.170740 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.170717 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:13:38.175871 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:38.175836 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1e0789-openshift-default-56fd556994-m6g5v"] Sep 10 03:13:39.005565 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:39.005520 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" path="/var/lib/kubelet/pods/e5b51ce6-59b5-4604-bbb4-68ab88bb3471/volumes" Sep 10 03:13:43.119782 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.119744 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:13:43.120336 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.120014 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" podUID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" containerName="maas-api" containerID="cri-o://44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0" gracePeriod=30 Sep 10 03:13:43.377978 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.377913 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:13:43.443157 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.443125 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68nq5\" (UniqueName: \"kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5\") pod \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " Sep 10 03:13:43.443352 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.443190 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca\") pod \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " Sep 10 03:13:43.443352 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.443230 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs\") pod \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " Sep 10 03:13:43.443352 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.443257 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls\") pod \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\" (UID: \"323ace4e-71d6-4dd0-9a94-72bb038d90e9\") " Sep 10 03:13:43.443632 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.443595 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "323ace4e-71d6-4dd0-9a94-72bb038d90e9" (UID: "323ace4e-71d6-4dd0-9a94-72bb038d90e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:13:43.445413 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.445384 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5" (OuterVolumeSpecName: "kube-api-access-68nq5") pod "323ace4e-71d6-4dd0-9a94-72bb038d90e9" (UID: "323ace4e-71d6-4dd0-9a94-72bb038d90e9"). InnerVolumeSpecName "kube-api-access-68nq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:43.445508 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.445420 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "323ace4e-71d6-4dd0-9a94-72bb038d90e9" (UID: "323ace4e-71d6-4dd0-9a94-72bb038d90e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:13:43.445508 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.445441 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "323ace4e-71d6-4dd0-9a94-72bb038d90e9" (UID: "323ace4e-71d6-4dd0-9a94-72bb038d90e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:13:43.544878 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.544827 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ace4e-71d6-4dd0-9a94-72bb038d90e9-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:43.544878 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.544881 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:43.545147 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.544899 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/323ace4e-71d6-4dd0-9a94-72bb038d90e9-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:43.545147 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:43.544912 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nq5\" (UniqueName: \"kubernetes.io/projected/323ace4e-71d6-4dd0-9a94-72bb038d90e9-kube-api-access-68nq5\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:44.187096 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.187056 2583 generic.go:356] "Generic (PLEG): container finished" podID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" containerID="44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0" exitCode=0 Sep 10 03:13:44.187601 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.187119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" Sep 10 03:13:44.187666 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.187140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" event={"ID":"323ace4e-71d6-4dd0-9a94-72bb038d90e9","Type":"ContainerDied","Data":"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0"} Sep 10 03:13:44.187723 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.187703 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc" event={"ID":"323ace4e-71d6-4dd0-9a94-72bb038d90e9","Type":"ContainerDied","Data":"b091a0e9417a06f2c9404193cf0a11a94f72b0a4805d18df8b348d1a57da0af4"} Sep 10 03:13:44.187770 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.187742 2583 scope.go:117] "RemoveContainer" containerID="44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0" Sep 10 03:13:44.197649 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.197630 2583 scope.go:117] "RemoveContainer" containerID="44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0" Sep 10 03:13:44.197996 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:44.197976 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0\": container with ID starting with 44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0 not found: ID does not exist" containerID="44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0" Sep 10 03:13:44.198066 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.198005 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0"} err="failed to get container status \"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0\": rpc error: code = NotFound desc = could not find container \"44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0\": container with ID starting with 44ba53f28ce2866ada871c2b2f63ea2ea72f53451f822fe4e564da3129bcd3c0 not found: ID does not exist" Sep 10 03:13:44.217135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.217106 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:13:44.222082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:44.222057 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c3058d-5b7db87d54-5jzcc"] Sep 10 03:13:45.007994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:45.007957 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" path="/var/lib/kubelet/pods/323ace4e-71d6-4dd0-9a94-72bb038d90e9/volumes" Sep 10 03:13:49.809331 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:49.809290 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:13:49.809822 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:49.809580 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" containerID="cri-o://30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723" gracePeriod=30 Sep 10 03:13:51.810757 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:51.810719 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 03:13:51.811172 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:51.810795 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 03:13:55.055487 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.055460 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:13:55.163971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.163881 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.163971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.163919 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.163971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.163944 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164256 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.163990 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164256 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164020 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjhsp\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164256 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164064 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164256 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164090 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164256 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164159 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164554 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164260 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket\") pod \"5895f088-2562-4117-b5ed-65dd2c658f2f\" (UID: \"5895f088-2562-4117-b5ed-65dd2c658f2f\") " Sep 10 03:13:55.164554 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164354 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:55.164554 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164475 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:55.164727 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164588 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:55.164727 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164664 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:55.164727 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164689 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:13:55.164960 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164938 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.165042 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164965 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.165042 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164982 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5895f088-2562-4117-b5ed-65dd2c658f2f-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.165042 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.164996 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.165042 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.165010 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.166305 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.166278 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:55.166428 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.166328 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:13:55.166428 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.166348 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:13:55.166428 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.166414 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp" (OuterVolumeSpecName: "kube-api-access-qjhsp") pod "5895f088-2562-4117-b5ed-65dd2c658f2f" (UID: "5895f088-2562-4117-b5ed-65dd2c658f2f"). InnerVolumeSpecName "kube-api-access-qjhsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:13:55.247962 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.247927 2583 generic.go:356] "Generic (PLEG): container finished" podID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerID="30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723" exitCode=0 Sep 10 03:13:55.247962 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.247969 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" Sep 10 03:13:55.248181 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.248011 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerDied","Data":"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723"} Sep 10 03:13:55.248181 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.248067 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h" event={"ID":"5895f088-2562-4117-b5ed-65dd2c658f2f","Type":"ContainerDied","Data":"c5a1334f6567dc2e45eb2d6b0f5106c0eef75d2a2325f35145a8abdad55412eb"} Sep 10 03:13:55.248181 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.248092 2583 scope.go:117] "RemoveContainer" containerID="30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723" Sep 10 03:13:55.258248 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.258230 2583 scope.go:117] "RemoveContainer" containerID="28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7" Sep 10 03:13:55.265504 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.265479 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.265504 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.265502 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.265661 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.265511 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5895f088-2562-4117-b5ed-65dd2c658f2f-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.265661 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.265520 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhsp\" (UniqueName: \"kubernetes.io/projected/5895f088-2562-4117-b5ed-65dd2c658f2f-kube-api-access-qjhsp\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:13:55.266748 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.266735 2583 scope.go:117] "RemoveContainer" containerID="30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723" Sep 10 03:13:55.267067 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:55.267049 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723\": container with ID starting with 30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723 not found: ID does not exist" containerID="30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723" Sep 10 03:13:55.267131 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.267076 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723"} err="failed to get container status \"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723\": rpc error: code = NotFound desc = could not find container \"30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723\": container with ID starting with 30a7c223d85f68ff0a3a5321e44a96b4ad135e91fc87a47bdda803df04e1d723 not found: ID does not exist" Sep 10 03:13:55.267131 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.267093 2583 scope.go:117] "RemoveContainer" containerID="28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7" Sep 10 03:13:55.267319 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:13:55.267302 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7\": container with ID starting with 28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7 not found: ID does not exist" containerID="28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7" Sep 10 03:13:55.267361 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.267323 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7"} err="failed to get container status \"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7\": rpc error: code = NotFound desc = could not find container \"28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7\": container with ID starting with 28339177ae94c9d5babb83a06d50b180c69dd0f766d1070f5772f18da51958c7 not found: ID does not exist" Sep 10 03:13:55.272486 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.272453 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:13:55.277032 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:55.277006 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c3058d-openshift-default-565b4fb5f8-hdt4h"] Sep 10 03:13:57.001965 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:13:57.001929 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" path="/var/lib/kubelet/pods/5895f088-2562-4117-b5ed-65dd2c658f2f/volumes" Sep 10 03:14:31.746185 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:31.746152 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:14:46.765600 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:46.765553 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" podUID="2892cbc8-c691-41b9-898e-a80eb19628ec" containerName="main" containerID="cri-o://563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7" gracePeriod=45 Sep 10 03:14:47.014561 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.014525 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:14:47.201343 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201249 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201343 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201303 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201399 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201456 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whj7x\" (UniqueName: \"kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201519 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201524 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:14:47.201585 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201546 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home\") pod \"2892cbc8-c691-41b9-898e-a80eb19628ec\" (UID: \"2892cbc8-c691-41b9-898e-a80eb19628ec\") " Sep 10 03:14:47.201906 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201705 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:14:47.201906 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201838 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home" (OuterVolumeSpecName: "home") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:14:47.202005 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201958 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-model-cache\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.202005 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.201977 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-tmp-dir\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.203707 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.203680 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x" (OuterVolumeSpecName: "kube-api-access-whj7x") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "kube-api-access-whj7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:14:47.203776 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.203686 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm" (OuterVolumeSpecName: "dshm") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:14:47.203818 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.203768 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2892cbc8-c691-41b9-898e-a80eb19628ec" (UID: "2892cbc8-c691-41b9-898e-a80eb19628ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:14:47.303353 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.303314 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-dshm\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.303353 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.303346 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whj7x\" (UniqueName: \"kubernetes.io/projected/2892cbc8-c691-41b9-898e-a80eb19628ec-kube-api-access-whj7x\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.303353 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.303357 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2892cbc8-c691-41b9-898e-a80eb19628ec-home\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.303604 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.303371 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2892cbc8-c691-41b9-898e-a80eb19628ec-tls-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:14:47.534237 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.534201 2583 generic.go:356] "Generic (PLEG): container finished" podID="2892cbc8-c691-41b9-898e-a80eb19628ec" containerID="563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7" exitCode=0 Sep 10 03:14:47.534418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.534240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" event={"ID":"2892cbc8-c691-41b9-898e-a80eb19628ec","Type":"ContainerDied","Data":"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7"} Sep 10 03:14:47.534418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.534248 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" Sep 10 03:14:47.534418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.534271 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch" event={"ID":"2892cbc8-c691-41b9-898e-a80eb19628ec","Type":"ContainerDied","Data":"d600df0abc2ff1e9be7152a1dd1cce7c04be0a91a1df9c092aa3310295648d3e"} Sep 10 03:14:47.534418 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.534287 2583 scope.go:117] "RemoveContainer" containerID="563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7" Sep 10 03:14:47.543453 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.543434 2583 scope.go:117] "RemoveContainer" containerID="563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7" Sep 10 03:14:47.543741 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:14:47.543720 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7\": container with ID starting with 563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7 not found: ID does not exist" containerID="563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7" Sep 10 03:14:47.543805 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.543750 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7"} err="failed to get container status \"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7\": rpc error: code = NotFound desc = could not find container \"563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7\": container with ID starting with 563a0264f24df441dd90dbb5293788fdd8ba250263148e11ad4f04cacb79a9d7 not found: ID does not exist" Sep 10 03:14:47.564695 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.564665 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:14:47.579889 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:47.579849 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/auth-test-model-7daa14-kserve-5555d945f-rcdch"] Sep 10 03:14:49.000311 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:14:49.000274 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2892cbc8-c691-41b9-898e-a80eb19628ec" path="/var/lib/kubelet/pods/2892cbc8-c691-41b9-898e-a80eb19628ec/volumes" Sep 10 03:15:37.300219 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.300181 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 03:15:37.300750 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.300420 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" podUID="89003daf-f265-429b-b174-c2d6cc0e5bd2" containerName="maas-api" containerID="cri-o://2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0" gracePeriod=30 Sep 10 03:15:37.768076 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.768049 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 03:15:37.795030 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.794998 2583 generic.go:356] "Generic (PLEG): container finished" podID="89003daf-f265-429b-b174-c2d6cc0e5bd2" containerID="2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0" exitCode=0 Sep 10 03:15:37.795198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.795044 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" Sep 10 03:15:37.795198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.795049 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" event={"ID":"89003daf-f265-429b-b174-c2d6cc0e5bd2","Type":"ContainerDied","Data":"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0"} Sep 10 03:15:37.795198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.795097 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q" event={"ID":"89003daf-f265-429b-b174-c2d6cc0e5bd2","Type":"ContainerDied","Data":"dbfecd78bddb865f93310630dbf1b6cd9d80f2ca546caddbe7dd3feeaefeecf0"} Sep 10 03:15:37.795198 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.795161 2583 scope.go:117] "RemoveContainer" containerID="2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0" Sep 10 03:15:37.804432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.804408 2583 scope.go:117] "RemoveContainer" containerID="2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0" Sep 10 03:15:37.804795 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:15:37.804768 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0\": container with ID starting with 2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0 not found: ID does not exist" containerID="2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0" Sep 10 03:15:37.804901 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.804809 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0"} err="failed to get container status \"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0\": rpc error: code = NotFound desc = could not find container \"2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0\": container with ID starting with 2a766cee71ea0cb0b8956fce179a4053615b2852677468915459c411a1a301f0 not found: ID does not exist" Sep 10 03:15:37.818808 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.818776 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") pod \"89003daf-f265-429b-b174-c2d6cc0e5bd2\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " Sep 10 03:15:37.818997 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.818877 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8\") pod \"89003daf-f265-429b-b174-c2d6cc0e5bd2\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " Sep 10 03:15:37.818997 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.818952 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca\") pod \"89003daf-f265-429b-b174-c2d6cc0e5bd2\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " Sep 10 03:15:37.818997 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.818980 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs\") pod \"89003daf-f265-429b-b174-c2d6cc0e5bd2\" (UID: \"89003daf-f265-429b-b174-c2d6cc0e5bd2\") " Sep 10 03:15:37.819329 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.819295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89003daf-f265-429b-b174-c2d6cc0e5bd2" (UID: "89003daf-f265-429b-b174-c2d6cc0e5bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:15:37.821388 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.821359 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89003daf-f265-429b-b174-c2d6cc0e5bd2" (UID: "89003daf-f265-429b-b174-c2d6cc0e5bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:15:37.821515 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.821501 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8" (OuterVolumeSpecName: "kube-api-access-csdc8") pod "89003daf-f265-429b-b174-c2d6cc0e5bd2" (UID: "89003daf-f265-429b-b174-c2d6cc0e5bd2"). InnerVolumeSpecName "kube-api-access-csdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:15:37.821587 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.821570 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89003daf-f265-429b-b174-c2d6cc0e5bd2" (UID: "89003daf-f265-429b-b174-c2d6cc0e5bd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:15:37.920357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.920315 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89003daf-f265-429b-b174-c2d6cc0e5bd2-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:15:37.920357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.920350 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:15:37.920357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.920361 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89003daf-f265-429b-b174-c2d6cc0e5bd2-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:15:37.920357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:37.920368 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/89003daf-f265-429b-b174-c2d6cc0e5bd2-kube-api-access-csdc8\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:15:38.120870 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:38.120819 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 03:15:38.124667 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:38.124637 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e6e6b8-6fcbd6b45-86q2q"] Sep 10 03:15:39.000354 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:15:39.000316 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89003daf-f265-429b-b174-c2d6cc0e5bd2" path="/var/lib/kubelet/pods/89003daf-f265-429b-b174-c2d6cc0e5bd2/volumes" Sep 10 03:16:37.121435 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:16:37.121407 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:16:37.132828 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:16:37.132801 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:17:31.401469 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.401426 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.401959 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.401976 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.401986 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.401994 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402004 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2892cbc8-c691-41b9-898e-a80eb19628ec" containerName="main" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402010 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2892cbc8-c691-41b9-898e-a80eb19628ec" containerName="main" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402026 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402032 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402041 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89003daf-f265-429b-b174-c2d6cc0e5bd2" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402046 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="89003daf-f265-429b-b174-c2d6cc0e5bd2" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402052 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be762343-05c9-4e7c-b4af-85b6c58071a2" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402057 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be762343-05c9-4e7c-b4af-85b6c58071a2" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402063 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402071 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402083 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b5504d-a5ab-4648-b239-5ee95322b03d" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402089 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b5504d-a5ab-4648-b239-5ee95322b03d" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402096 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402101 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402106 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402111 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402117 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a877c31a-d5a6-4317-88a8-e755893cc8eb" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402122 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a877c31a-d5a6-4317-88a8-e755893cc8eb" containerName="maas-api" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402129 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.402123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402134 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402191 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b5504d-a5ab-4648-b239-5ee95322b03d" containerName="maas-api" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402200 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be762343-05c9-4e7c-b4af-85b6c58071a2" containerName="maas-api" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402207 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402213 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402220 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="89003daf-f265-429b-b174-c2d6cc0e5bd2" containerName="maas-api" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402229 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b51ce6-59b5-4604-bbb4-68ab88bb3471" containerName="istio-proxy" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402238 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402246 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="323ace4e-71d6-4dd0-9a94-72bb038d90e9" containerName="maas-api" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402258 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a877c31a-d5a6-4317-88a8-e755893cc8eb" containerName="maas-api" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402265 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2892cbc8-c691-41b9-898e-a80eb19628ec" containerName="main" Sep 10 03:17:31.403234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.402273 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab26a95-d2c6-453d-a366-20f8e2a4d24d" containerName="istio-proxy" Sep 10 03:17:31.405220 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.405199 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.407189 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.407166 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c9cef5-openshift-default-dockercfg-sf6bn\"" Sep 10 03:17:31.414528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.414499 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:17:31.448446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkx6\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448561 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.448737 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448708 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.449018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.449018 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.448793 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.549957 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550339 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.549970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkx6\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550339 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.550255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550339 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.550278 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550339 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.550313 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550498 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.550353 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.550799 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.550782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.552100 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.552075 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.552345 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.552323 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.567239 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.567200 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.567359 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.567301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkx6\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6\") pod \"e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.717333 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.717220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:31.996025 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:31.996000 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:17:31.998118 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:17:31.998086 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e90603_5645_4997_a39b_0e690bedf0b3.slice/crio-48a453a834aaf19404b63677b1d63ae6f2ccb6b53534bf0a569041524e567977 WatchSource:0}: Error finding container 48a453a834aaf19404b63677b1d63ae6f2ccb6b53534bf0a569041524e567977: Status 404 returned error can't find the container with id 48a453a834aaf19404b63677b1d63ae6f2ccb6b53534bf0a569041524e567977 Sep 10 03:17:32.000341 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.000305 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:17:32.000414 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.000374 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:17:32.000414 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.000404 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:17:32.389542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.389504 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerStarted","Data":"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e"} Sep 10 03:17:32.389725 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.389566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerStarted","Data":"48a453a834aaf19404b63677b1d63ae6f2ccb6b53534bf0a569041524e567977"} Sep 10 03:17:32.408258 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.408196 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podStartSLOduration=1.408179154 podStartE2EDuration="1.408179154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:17:32.406982792 +0000 UTC m=+1855.930839477" watchObservedRunningTime="2026-09-10 03:17:32.408179154 +0000 UTC m=+1855.932035841" Sep 10 03:17:32.718054 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:32.717954 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:33.723247 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:33.723208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:34.400204 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:34.400173 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:17:34.401294 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:17:34.401276 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:18:05.442051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:05.442005 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 03:18:05.442527 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:05.442096 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 03:18:06.575881 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.575823 2583 generic.go:356] "Generic (PLEG): container finished" podID="21e90603-5645-4997-a39b-0e690bedf0b3" containerID="65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e" exitCode=0 Sep 10 03:18:06.576334 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.575898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerDied","Data":"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e"} Sep 10 03:18:06.576385 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.576332 2583 scope.go:117] "RemoveContainer" containerID="65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e" Sep 10 03:18:06.577472 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.577440 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:18:06.577540 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.577511 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:18:06.577577 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:06.577545 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:18:07.583360 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:07.583320 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerStarted","Data":"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c"} Sep 10 03:18:07.718053 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:07.718010 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:18:07.718271 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:07.718246 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:07.718321 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:07.718297 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:08.718587 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:08.718552 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:08.719008 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:08.718627 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:09.718712 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:09.718675 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:09.719164 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:09.718736 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:10.718060 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:10.718022 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:10.718265 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:10.718086 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:11.718513 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:11.718464 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:11.718956 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:11.718531 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:12.717831 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:12.717774 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 03:18:12.718042 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:12.717924 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 03:18:13.721308 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:13.721275 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:18:13.721726 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:13.721701 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:18:13.722525 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:18:13.722504 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:19:10.306880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.306826 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:10.307447 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.307381 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5895f088-2562-4117-b5ed-65dd2c658f2f" containerName="istio-proxy" Sep 10 03:19:10.310507 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.310487 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.312692 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.312670 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-be281e-openshift-default-dockercfg-vttp8\"" Sep 10 03:19:10.321634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.321607 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:10.445084 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445217 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445301 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.445528 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.445443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjng\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546308 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjng\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546536 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546536 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546716 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546837 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546837 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546837 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.546837 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.547082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546853 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.547082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.546971 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.547082 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.547034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.547467 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.547442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.548951 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.548927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.549325 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.549308 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.554378 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.554357 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.554593 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.554573 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjng\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng\") pod \"e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.622824 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.622714 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:10.909041 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.909015 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:10.910665 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:19:10.910637 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6bd527_1966_4e18_9fb4_5be2d236ce3c.slice/crio-a68e7fe9a7df3f266b3450654aeb9b61d64e0e62fa0b217ae0114af0d5de4869 WatchSource:0}: Error finding container a68e7fe9a7df3f266b3450654aeb9b61d64e0e62fa0b217ae0114af0d5de4869: Status 404 returned error can't find the container with id a68e7fe9a7df3f266b3450654aeb9b61d64e0e62fa0b217ae0114af0d5de4869 Sep 10 03:19:10.913190 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.913155 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:10.913270 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.913229 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:10.913270 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.913258 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:10.927469 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:10.927440 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerStarted","Data":"a68e7fe9a7df3f266b3450654aeb9b61d64e0e62fa0b217ae0114af0d5de4869"} Sep 10 03:19:11.934021 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:11.933981 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerStarted","Data":"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e"} Sep 10 03:19:11.952705 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:11.952649 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podStartSLOduration=1.9526300060000001 podStartE2EDuration="1.952630006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:19:11.951620942 +0000 UTC m=+1955.475477660" watchObservedRunningTime="2026-09-10 03:19:11.952630006 +0000 UTC m=+1955.476486693" Sep 10 03:19:12.623266 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:12.623229 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:12.628703 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:12.628677 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:12.939748 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:12.939673 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:12.940818 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:12.940797 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:36.070003 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.069965 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerID="6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e" exitCode=0 Sep 10 03:19:36.070551 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.070018 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerDied","Data":"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e"} Sep 10 03:19:36.070551 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.070535 2583 scope.go:117] "RemoveContainer" containerID="6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e" Sep 10 03:19:36.071754 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.071725 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:36.071840 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.071795 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:36.071840 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:36.071827 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 10 03:19:37.078659 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:37.078618 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerStarted","Data":"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5"} Sep 10 03:19:37.623261 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:37.623218 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:37.623518 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:37.623489 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 03:19:37.623579 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:37.623558 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 03:19:38.623264 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:38.623225 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 03:19:38.623678 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:38.623288 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 03:19:39.623620 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:39.623587 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 03:19:39.624048 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:39.623642 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 03:19:40.623897 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:40.623838 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 03:19:40.624320 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:40.623920 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 03:19:41.623971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:41.623933 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 03:19:41.624475 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:41.624036 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 03:19:42.635675 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:42.635643 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:42.636142 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:42.635970 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:42.637012 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:42.636985 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:52.217031 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:52.216995 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:52.217424 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:52.217196 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" containerID="cri-o://3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5" gracePeriod=30 Sep 10 03:19:57.482539 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.482474 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:57.556171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556143 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556179 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556205 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556241 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccjng\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556446 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556371 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556460 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556493 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket\") pod \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\" (UID: \"0b6bd527-1966-4e18-9fb4-5be2d236ce3c\") " Sep 10 03:19:57.556732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556601 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:19:57.556732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556626 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:19:57.556960 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556834 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:19:57.556960 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556852 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:19:57.557081 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.556985 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.557081 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.557009 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.557081 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.557023 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.557081 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.557035 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.557308 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.557116 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:19:57.558745 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.558707 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:19:57.558887 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.558790 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:19:57.558946 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.558909 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng" (OuterVolumeSpecName: "kube-api-access-ccjng") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "kube-api-access-ccjng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:19:57.559113 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.559082 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b6bd527-1966-4e18-9fb4-5be2d236ce3c" (UID: "0b6bd527-1966-4e18-9fb4-5be2d236ce3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:19:57.658227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.658189 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.658227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.658222 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.658227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.658231 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.658227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.658239 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:57.658494 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:57.658249 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjng\" (UniqueName: \"kubernetes.io/projected/0b6bd527-1966-4e18-9fb4-5be2d236ce3c-kube-api-access-ccjng\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:19:58.201186 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.201152 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerID="3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5" exitCode=0 Sep 10 03:19:58.201372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.201207 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" Sep 10 03:19:58.201372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.201235 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerDied","Data":"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5"} Sep 10 03:19:58.201372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.201295 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh" event={"ID":"0b6bd527-1966-4e18-9fb4-5be2d236ce3c","Type":"ContainerDied","Data":"a68e7fe9a7df3f266b3450654aeb9b61d64e0e62fa0b217ae0114af0d5de4869"} Sep 10 03:19:58.201372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.201318 2583 scope.go:117] "RemoveContainer" containerID="3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5" Sep 10 03:19:58.218596 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.218573 2583 scope.go:117] "RemoveContainer" containerID="6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e" Sep 10 03:19:58.227456 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.227432 2583 scope.go:117] "RemoveContainer" containerID="3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5" Sep 10 03:19:58.227776 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:19:58.227749 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5\": container with ID starting with 3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5 not found: ID does not exist" containerID="3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5" Sep 10 03:19:58.227844 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.227785 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5"} err="failed to get container status \"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5\": rpc error: code = NotFound desc = could not find container \"3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5\": container with ID starting with 3b85354948a4d079071282421afd7eb218f03f1e1fddb508f0609e94ca52ffd5 not found: ID does not exist" Sep 10 03:19:58.227844 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.227805 2583 scope.go:117] "RemoveContainer" containerID="6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e" Sep 10 03:19:58.228122 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:19:58.228105 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e\": container with ID starting with 6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e not found: ID does not exist" containerID="6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e" Sep 10 03:19:58.228161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.228126 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e"} err="failed to get container status \"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e\": rpc error: code = NotFound desc = could not find container \"6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e\": container with ID starting with 6d284b430c338fb040afd2bb04016dbbfd5c979ea972cc813b5ec981beb1430e not found: ID does not exist" Sep 10 03:19:58.228498 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.228480 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:58.233811 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:58.233785 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-be281e-openshift-default-5c4dbb5d5d-pzxjh"] Sep 10 03:19:59.000348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:19:59.000312 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" path="/var/lib/kubelet/pods/0b6bd527-1966-4e18-9fb4-5be2d236ce3c/volumes" Sep 10 03:21:00.007371 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:00.007283 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:21:00.007955 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:00.007596 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" containerID="cri-o://7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c" gracePeriod=30 Sep 10 03:21:05.265658 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.265632 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:21:05.443744 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443632 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.443744 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443747 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443771 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443839 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443923 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443955 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdkx6\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444034 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.443976 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444286 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444035 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444286 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444060 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs\") pod \"21e90603-5645-4997-a39b-0e690bedf0b3\" (UID: \"21e90603-5645-4997-a39b-0e690bedf0b3\") " Sep 10 03:21:05.444286 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444146 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:05.444286 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444169 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:05.444513 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444415 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:05.444576 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444563 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.444634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444583 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.444634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444597 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.444735 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444636 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:21:05.444735 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.444696 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:05.446101 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.446076 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:05.446211 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.446114 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:21:05.446320 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.446301 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:05.446410 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.446391 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6" (OuterVolumeSpecName: "kube-api-access-zdkx6") pod "21e90603-5645-4997-a39b-0e690bedf0b3" (UID: "21e90603-5645-4997-a39b-0e690bedf0b3"). InnerVolumeSpecName "kube-api-access-zdkx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:05.545980 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.545944 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e90603-5645-4997-a39b-0e690bedf0b3-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.545980 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.545973 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.545980 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.545984 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkx6\" (UniqueName: \"kubernetes.io/projected/21e90603-5645-4997-a39b-0e690bedf0b3-kube-api-access-zdkx6\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.546230 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.545994 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e90603-5645-4997-a39b-0e690bedf0b3-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.546230 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.546003 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.546230 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.546011 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e90603-5645-4997-a39b-0e690bedf0b3-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:05.552732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.552698 2583 generic.go:356] "Generic (PLEG): container finished" podID="21e90603-5645-4997-a39b-0e690bedf0b3" containerID="7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c" exitCode=0 Sep 10 03:21:05.552918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.552747 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" Sep 10 03:21:05.552918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.552783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerDied","Data":"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c"} Sep 10 03:21:05.552918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.552839 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw" event={"ID":"21e90603-5645-4997-a39b-0e690bedf0b3","Type":"ContainerDied","Data":"48a453a834aaf19404b63677b1d63ae6f2ccb6b53534bf0a569041524e567977"} Sep 10 03:21:05.552918 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.552893 2583 scope.go:117] "RemoveContainer" containerID="7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c" Sep 10 03:21:05.562340 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.562322 2583 scope.go:117] "RemoveContainer" containerID="65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e" Sep 10 03:21:05.570685 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.570666 2583 scope.go:117] "RemoveContainer" containerID="7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c" Sep 10 03:21:05.571004 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:05.570983 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c\": container with ID starting with 7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c not found: ID does not exist" containerID="7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c" Sep 10 03:21:05.571054 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.571014 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c"} err="failed to get container status \"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c\": rpc error: code = NotFound desc = could not find container \"7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c\": container with ID starting with 7d98690da4706a29322bfd469f2170b382ff94f09b70980af67b2fde0b1d015c not found: ID does not exist" Sep 10 03:21:05.571054 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.571034 2583 scope.go:117] "RemoveContainer" containerID="65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e" Sep 10 03:21:05.571427 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:05.571326 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e\": container with ID starting with 65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e not found: ID does not exist" containerID="65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e" Sep 10 03:21:05.571427 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.571366 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e"} err="failed to get container status \"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e\": rpc error: code = NotFound desc = could not find container \"65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e\": container with ID starting with 65991b857cbc4cc34f111885c30f2927363140d1726572486bd3b402fdec1d7e not found: ID does not exist" Sep 10 03:21:05.574413 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.574388 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:21:05.578186 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:05.578159 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c9cef5-openshift-default-8dbf59dc6-jtscw"] Sep 10 03:21:07.000810 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:07.000777 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" path="/var/lib/kubelet/pods/21e90603-5645-4997-a39b-0e690bedf0b3/volumes" Sep 10 03:21:28.581522 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.581484 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:21:28.582494 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.581799 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" podUID="0597dcb9-158f-4799-b71f-15ac02ebfe57" containerName="maas-api" containerID="cri-o://291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294" gracePeriod=30 Sep 10 03:21:28.836455 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.836382 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:21:28.968201 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968164 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs\") pod \"0597dcb9-158f-4799-b71f-15ac02ebfe57\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " Sep 10 03:21:28.968426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968240 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lfs\" (UniqueName: \"kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs\") pod \"0597dcb9-158f-4799-b71f-15ac02ebfe57\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " Sep 10 03:21:28.968426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968290 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca\") pod \"0597dcb9-158f-4799-b71f-15ac02ebfe57\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " Sep 10 03:21:28.968426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968356 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") pod \"0597dcb9-158f-4799-b71f-15ac02ebfe57\" (UID: \"0597dcb9-158f-4799-b71f-15ac02ebfe57\") " Sep 10 03:21:28.968693 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968671 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0597dcb9-158f-4799-b71f-15ac02ebfe57" (UID: "0597dcb9-158f-4799-b71f-15ac02ebfe57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:21:28.968911 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.968890 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0597dcb9-158f-4799-b71f-15ac02ebfe57-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:28.970651 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.970624 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0597dcb9-158f-4799-b71f-15ac02ebfe57" (UID: "0597dcb9-158f-4799-b71f-15ac02ebfe57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:28.970775 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.970646 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0597dcb9-158f-4799-b71f-15ac02ebfe57" (UID: "0597dcb9-158f-4799-b71f-15ac02ebfe57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:28.970775 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:28.970738 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs" (OuterVolumeSpecName: "kube-api-access-97lfs") pod "0597dcb9-158f-4799-b71f-15ac02ebfe57" (UID: "0597dcb9-158f-4799-b71f-15ac02ebfe57"). InnerVolumeSpecName "kube-api-access-97lfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:29.069701 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.069664 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:29.069701 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.069695 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0597dcb9-158f-4799-b71f-15ac02ebfe57-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:29.069701 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.069705 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lfs\" (UniqueName: \"kubernetes.io/projected/0597dcb9-158f-4799-b71f-15ac02ebfe57-kube-api-access-97lfs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:29.678768 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.678734 2583 generic.go:356] "Generic (PLEG): container finished" podID="0597dcb9-158f-4799-b71f-15ac02ebfe57" containerID="291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294" exitCode=0 Sep 10 03:21:29.679318 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.678788 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" Sep 10 03:21:29.679318 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.678826 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" event={"ID":"0597dcb9-158f-4799-b71f-15ac02ebfe57","Type":"ContainerDied","Data":"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294"} Sep 10 03:21:29.679318 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.678904 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r" event={"ID":"0597dcb9-158f-4799-b71f-15ac02ebfe57","Type":"ContainerDied","Data":"188165a7890ede9bd9e69298e0f3b7386ea5c682a1af1260923efd5a65eb1aa2"} Sep 10 03:21:29.679318 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.678926 2583 scope.go:117] "RemoveContainer" containerID="291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294" Sep 10 03:21:29.687639 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.687622 2583 scope.go:117] "RemoveContainer" containerID="291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294" Sep 10 03:21:29.687941 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:29.687923 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294\": container with ID starting with 291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294 not found: ID does not exist" containerID="291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294" Sep 10 03:21:29.687991 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.687951 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294"} err="failed to get container status \"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294\": rpc error: code = NotFound desc = could not find container \"291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294\": container with ID starting with 291bdbd0994659ac296a929d083d5f3730f461694cb6aac71f19cafbcd8f4294 not found: ID does not exist" Sep 10 03:21:29.694426 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.694396 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:21:29.698932 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:29.698907 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7727-66ff67976c-gt94r"] Sep 10 03:21:31.000531 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:31.000496 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0597dcb9-158f-4799-b71f-15ac02ebfe57" path="/var/lib/kubelet/pods/0597dcb9-158f-4799-b71f-15ac02ebfe57/volumes" Sep 10 03:21:35.318613 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:35.318577 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:21:35.319181 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:35.318889 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" containerID="cri-o://34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903" gracePeriod=30 Sep 10 03:21:35.837685 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:35.837641 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 03:21:35.837874 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:35.837711 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 03:21:37.154187 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:37.154155 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:21:37.166562 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:37.166536 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:21:40.576011 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.575983 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:21:40.686615 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686525 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686615 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686564 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686615 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686593 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686638 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686665 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686717 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686810 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686837 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:40.686939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686848 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686916 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686923 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert\") pod \"ae7136d1-f59e-4948-b707-559b0a10b37b\" (UID: \"ae7136d1-f59e-4948-b707-559b0a10b37b\") " Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686980 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.686998 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.687233 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:21:40.687316 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.687297 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.687542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.687325 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-data\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.687542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.687341 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-workload-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.687542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.687354 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-credential-socket\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.689067 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.689046 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk" (OuterVolumeSpecName: "kube-api-access-7ctbk") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "kube-api-access-7ctbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:40.689146 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.689069 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 03:21:40.689350 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.689332 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:40.689397 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.689349 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae7136d1-f59e-4948-b707-559b0a10b37b" (UID: "ae7136d1-f59e-4948-b707-559b0a10b37b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 03:21:40.743674 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.743631 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerID="34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903" exitCode=0 Sep 10 03:21:40.743853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.743707 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" Sep 10 03:21:40.743853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.743748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerDied","Data":"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903"} Sep 10 03:21:40.743853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.743800 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn" event={"ID":"ae7136d1-f59e-4948-b707-559b0a10b37b","Type":"ContainerDied","Data":"d8875ec1869ca79203063e98b71ad723f75105d340926718c876d619774e24ab"} Sep 10 03:21:40.743853 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.743824 2583 scope.go:117] "RemoveContainer" containerID="34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903" Sep 10 03:21:40.753638 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.753615 2583 scope.go:117] "RemoveContainer" containerID="f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c" Sep 10 03:21:40.762070 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.762051 2583 scope.go:117] "RemoveContainer" containerID="34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903" Sep 10 03:21:40.762373 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:40.762354 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903\": container with ID starting with 34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903 not found: ID does not exist" containerID="34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903" Sep 10 03:21:40.762421 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.762383 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903"} err="failed to get container status \"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903\": rpc error: code = NotFound desc = could not find container \"34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903\": container with ID starting with 34866925e4e4e6853f5ef9c78c9636f989b5434c948336a94504fe37076b3903 not found: ID does not exist" Sep 10 03:21:40.762421 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.762400 2583 scope.go:117] "RemoveContainer" containerID="f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c" Sep 10 03:21:40.762666 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:40.762644 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c\": container with ID starting with f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c not found: ID does not exist" containerID="f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c" Sep 10 03:21:40.762727 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.762678 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c"} err="failed to get container status \"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c\": rpc error: code = NotFound desc = could not find container \"f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c\": container with ID starting with f4513fb7029cf9a4d2edaeb3050d294f0a79208f6d0b84d8ec2d0a2954730e5c not found: ID does not exist" Sep 10 03:21:40.770711 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.770676 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:21:40.778749 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.778713 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-af7727-openshift-default-86dd8b9564-9qbcn"] Sep 10 03:21:40.788827 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.788800 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-envoy\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.788827 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.788828 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-podinfo\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.788994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.788837 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7136d1-f59e-4948-b707-559b0a10b37b-istiod-ca-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.788994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.788849 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-istio-token\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:40.788994 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:40.788870 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ae7136d1-f59e-4948-b707-559b0a10b37b-kube-api-access-7ctbk\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:41.000931 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:41.000898 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" path="/var/lib/kubelet/pods/ae7136d1-f59e-4948-b707-559b0a10b37b/volumes" Sep 10 03:21:46.164575 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.164537 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:21:46.165027 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.164760 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" podUID="858e762c-c89e-44b6-9175-c9e6d4867710" containerName="maas-api" containerID="cri-o://f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d" gracePeriod=30 Sep 10 03:21:46.412943 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.412916 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:21:46.540458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.540423 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca\") pod \"858e762c-c89e-44b6-9175-c9e6d4867710\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " Sep 10 03:21:46.540668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.540532 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") pod \"858e762c-c89e-44b6-9175-c9e6d4867710\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " Sep 10 03:21:46.540668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.540565 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55dks\" (UniqueName: \"kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks\") pod \"858e762c-c89e-44b6-9175-c9e6d4867710\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " Sep 10 03:21:46.540790 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.540667 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs\") pod \"858e762c-c89e-44b6-9175-c9e6d4867710\" (UID: \"858e762c-c89e-44b6-9175-c9e6d4867710\") " Sep 10 03:21:46.540873 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.540780 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "858e762c-c89e-44b6-9175-c9e6d4867710" (UID: "858e762c-c89e-44b6-9175-c9e6d4867710"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:21:46.541061 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.541039 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/858e762c-c89e-44b6-9175-c9e6d4867710-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:46.542883 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.542839 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "858e762c-c89e-44b6-9175-c9e6d4867710" (UID: "858e762c-c89e-44b6-9175-c9e6d4867710"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:46.543040 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.543014 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks" (OuterVolumeSpecName: "kube-api-access-55dks") pod "858e762c-c89e-44b6-9175-c9e6d4867710" (UID: "858e762c-c89e-44b6-9175-c9e6d4867710"). InnerVolumeSpecName "kube-api-access-55dks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:46.543104 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.543065 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "858e762c-c89e-44b6-9175-c9e6d4867710" (UID: "858e762c-c89e-44b6-9175-c9e6d4867710"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:46.641971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.641928 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:46.641971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.641968 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55dks\" (UniqueName: \"kubernetes.io/projected/858e762c-c89e-44b6-9175-c9e6d4867710-kube-api-access-55dks\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:46.641971 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.641978 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858e762c-c89e-44b6-9175-c9e6d4867710-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:46.775216 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.775177 2583 generic.go:356] "Generic (PLEG): container finished" podID="858e762c-c89e-44b6-9175-c9e6d4867710" containerID="f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d" exitCode=0 Sep 10 03:21:46.775387 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.775225 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" Sep 10 03:21:46.775387 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.775267 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" event={"ID":"858e762c-c89e-44b6-9175-c9e6d4867710","Type":"ContainerDied","Data":"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d"} Sep 10 03:21:46.775387 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.775319 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh" event={"ID":"858e762c-c89e-44b6-9175-c9e6d4867710","Type":"ContainerDied","Data":"ee586ba6fa7057dbbe32b0d9637b70904c6f7a3c84ad99424f282ac8bc8d42a2"} Sep 10 03:21:46.775387 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.775345 2583 scope.go:117] "RemoveContainer" containerID="f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d" Sep 10 03:21:46.784789 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.784771 2583 scope.go:117] "RemoveContainer" containerID="f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d" Sep 10 03:21:46.785097 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:46.785078 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d\": container with ID starting with f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d not found: ID does not exist" containerID="f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d" Sep 10 03:21:46.785161 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.785105 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d"} err="failed to get container status \"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d\": rpc error: code = NotFound desc = could not find container \"f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d\": container with ID starting with f171d8a8f78977e2bc6d020d48605fe00de78eaa2748aaefddebe037169ee74d not found: ID does not exist" Sep 10 03:21:46.795309 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.795245 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:21:46.799170 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:46.799144 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7daa14-7fd66745cd-t86mh"] Sep 10 03:21:47.000264 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:47.000232 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858e762c-c89e-44b6-9175-c9e6d4867710" path="/var/lib/kubelet/pods/858e762c-c89e-44b6-9175-c9e6d4867710/volumes" Sep 10 03:21:55.567273 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.567235 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 03:21:55.567788 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.567471 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76d967fdff-gj4zj" podUID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" containerName="manager" containerID="cri-o://6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4" gracePeriod=30 Sep 10 03:21:55.818964 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.818901 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 03:21:55.820922 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.820892 2583 generic.go:356] "Generic (PLEG): container finished" podID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" containerID="6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4" exitCode=0 Sep 10 03:21:55.821045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.820941 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-gj4zj" Sep 10 03:21:55.821045 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.820976 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-gj4zj" event={"ID":"dc460ea6-63d7-48d5-b0c8-d85a7942e884","Type":"ContainerDied","Data":"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4"} Sep 10 03:21:55.821134 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.821051 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-gj4zj" event={"ID":"dc460ea6-63d7-48d5-b0c8-d85a7942e884","Type":"ContainerDied","Data":"6b6bdc63c8aadfd650dcfb9e745157c1a1068631fd98f44760e381ee9d06836f"} Sep 10 03:21:55.821134 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.821075 2583 scope.go:117] "RemoveContainer" containerID="6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4" Sep 10 03:21:55.829894 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.829852 2583 scope.go:117] "RemoveContainer" containerID="6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4" Sep 10 03:21:55.830196 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:21:55.830179 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4\": container with ID starting with 6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4 not found: ID does not exist" containerID="6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4" Sep 10 03:21:55.830265 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.830204 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4"} err="failed to get container status \"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4\": rpc error: code = NotFound desc = could not find container \"6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4\": container with ID starting with 6038b7e666781824ad9ebecd550f3a7ca0e6c35148e8c25129f5c53ce60210b4 not found: ID does not exist" Sep 10 03:21:55.830379 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.830363 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs\") pod \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " Sep 10 03:21:55.830416 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.830394 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5t98\" (UniqueName: \"kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98\") pod \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " Sep 10 03:21:55.830465 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.830454 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert\") pod \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\" (UID: \"dc460ea6-63d7-48d5-b0c8-d85a7942e884\") " Sep 10 03:21:55.832402 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.832375 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc460ea6-63d7-48d5-b0c8-d85a7942e884" (UID: "dc460ea6-63d7-48d5-b0c8-d85a7942e884"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:55.832517 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.832426 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98" (OuterVolumeSpecName: "kube-api-access-p5t98") pod "dc460ea6-63d7-48d5-b0c8-d85a7942e884" (UID: "dc460ea6-63d7-48d5-b0c8-d85a7942e884"). InnerVolumeSpecName "kube-api-access-p5t98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:21:55.832517 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.832471 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc460ea6-63d7-48d5-b0c8-d85a7942e884" (UID: "dc460ea6-63d7-48d5-b0c8-d85a7942e884"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:21:55.931246 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.931211 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:55.931246 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.931239 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5t98\" (UniqueName: \"kubernetes.io/projected/dc460ea6-63d7-48d5-b0c8-d85a7942e884-kube-api-access-p5t98\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:55.931246 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:55.931249 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc460ea6-63d7-48d5-b0c8-d85a7942e884-webhook-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:21:56.142444 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:56.142400 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 03:21:56.147309 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:56.147274 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76d967fdff-gj4zj"] Sep 10 03:21:57.000398 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:21:57.000359 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" path="/var/lib/kubelet/pods/dc460ea6-63d7-48d5-b0c8-d85a7942e884/volumes" Sep 10 03:22:56.941751 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.941714 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942254 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942274 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942289 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942299 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942309 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942318 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942333 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942341 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942353 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858e762c-c89e-44b6-9175-c9e6d4867710" containerName="maas-api" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942361 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="858e762c-c89e-44b6-9175-c9e6d4867710" containerName="maas-api" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942372 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0597dcb9-158f-4799-b71f-15ac02ebfe57" containerName="maas-api" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942381 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0597dcb9-158f-4799-b71f-15ac02ebfe57" containerName="maas-api" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942389 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" containerName="manager" Sep 10 03:22:56.942399 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942397 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" containerName="manager" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942418 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942426 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942520 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942533 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942542 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942554 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942563 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc460ea6-63d7-48d5-b0c8-d85a7942e884" containerName="manager" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942573 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="858e762c-c89e-44b6-9175-c9e6d4867710" containerName="maas-api" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942583 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7136d1-f59e-4948-b707-559b0a10b37b" containerName="istio-proxy" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942592 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0597dcb9-158f-4799-b71f-15ac02ebfe57" containerName="maas-api" Sep 10 03:22:56.943178 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.942599 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e90603-5645-4997-a39b-0e690bedf0b3" containerName="istio-proxy" Sep 10 03:22:56.946079 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.946060 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:56.948111 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.948089 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 03:22:56.949065 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.948947 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 03:22:56.949065 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.949045 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lg6lz\"" Sep 10 03:22:56.949234 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.949161 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 03:22:56.955008 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:56.954987 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:22:57.027812 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.027773 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.028017 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.027837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.028017 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.027938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tbd\" (UniqueName: \"kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.128384 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.128350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.128581 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.128425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tbd\" (UniqueName: \"kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.128581 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.128470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.130901 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.130852 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.130901 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.130898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.135642 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.135615 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tbd\" (UniqueName: \"kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd\") pod \"maas-controller-76d967fdff-sfnpx\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.257733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.257708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:57.527634 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.527563 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:22:57.530572 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:22:57.530542 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6072bcb_7b7b_4b7e_abc4_ee529727c7b0.slice/crio-5e34933c665d77229a89800c55bd0af794dede47b594fb086a6a0b62716251da WatchSource:0}: Error finding container 5e34933c665d77229a89800c55bd0af794dede47b594fb086a6a0b62716251da: Status 404 returned error can't find the container with id 5e34933c665d77229a89800c55bd0af794dede47b594fb086a6a0b62716251da Sep 10 03:22:57.532035 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:57.532016 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 03:22:58.130200 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:58.130163 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-sfnpx" event={"ID":"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0","Type":"ContainerStarted","Data":"5e34933c665d77229a89800c55bd0af794dede47b594fb086a6a0b62716251da"} Sep 10 03:22:59.137074 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:59.137029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-sfnpx" event={"ID":"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0","Type":"ContainerStarted","Data":"6ce00e5155b164771bf3281a61ccb2b2065a587233f2cdc42327e9e31e6c2617"} Sep 10 03:22:59.137501 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:59.137172 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:22:59.155005 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:22:59.154952 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76d967fdff-sfnpx" podStartSLOduration=2.470641384 podStartE2EDuration="3.154935549s" totalImagesPullingTime="684.294165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:22:59.152660782 +0000 UTC m=+2182.676517472" watchObservedRunningTime="2026-09-10 03:22:59.154935549 +0000 UTC m=+2182.678792235" Sep 10 03:23:10.146244 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:10.146208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:23:19.566880 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:19.566816 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:23:19.567350 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:19.567063 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76d967fdff-sfnpx" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerName="manager" containerID="cri-o://6ce00e5155b164771bf3281a61ccb2b2065a587233f2cdc42327e9e31e6c2617" gracePeriod=30 Sep 10 03:23:20.143160 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.143122 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-76d967fdff-sfnpx" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.71:8081/readyz\": dial tcp 10.132.0.71:8081: connect: connection refused" Sep 10 03:23:20.244063 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.244015 2583 generic.go:356] "Generic (PLEG): container finished" podID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerID="6ce00e5155b164771bf3281a61ccb2b2065a587233f2cdc42327e9e31e6c2617" exitCode=0 Sep 10 03:23:20.244253 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.244081 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-sfnpx" event={"ID":"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0","Type":"ContainerDied","Data":"6ce00e5155b164771bf3281a61ccb2b2065a587233f2cdc42327e9e31e6c2617"} Sep 10 03:23:20.309351 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.309323 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:23:20.340205 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.340173 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7tbd\" (UniqueName: \"kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd\") pod \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " Sep 10 03:23:20.340420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.340240 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert\") pod \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " Sep 10 03:23:20.340420 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.340411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs\") pod \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\" (UID: \"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0\") " Sep 10 03:23:20.342462 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.342426 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd" (OuterVolumeSpecName: "kube-api-access-w7tbd") pod "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" (UID: "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0"). InnerVolumeSpecName "kube-api-access-w7tbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:23:20.342605 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.342432 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" (UID: "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:23:20.342886 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.342843 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" (UID: "d6072bcb-7b7b-4b7e-abc4-ee529727c7b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:23:20.442282 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.442195 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:23:20.442282 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.442229 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7tbd\" (UniqueName: \"kubernetes.io/projected/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-kube-api-access-w7tbd\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:23:20.442282 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:20.442240 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0-webhook-cert\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:23:21.250409 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:21.250376 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-sfnpx" Sep 10 03:23:21.250409 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:21.250390 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-sfnpx" event={"ID":"d6072bcb-7b7b-4b7e-abc4-ee529727c7b0","Type":"ContainerDied","Data":"5e34933c665d77229a89800c55bd0af794dede47b594fb086a6a0b62716251da"} Sep 10 03:23:21.250976 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:21.250451 2583 scope.go:117] "RemoveContainer" containerID="6ce00e5155b164771bf3281a61ccb2b2065a587233f2cdc42327e9e31e6c2617" Sep 10 03:23:21.266392 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:21.266360 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:23:21.268202 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:21.268181 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76d967fdff-sfnpx"] Sep 10 03:23:22.024044 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024009 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76d967fdff-xvg8l"] Sep 10 03:23:22.024440 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024422 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:23:22.024537 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024442 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6bd527-1966-4e18-9fb4-5be2d236ce3c" containerName="istio-proxy" Sep 10 03:23:22.024537 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024455 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerName="manager" Sep 10 03:23:22.024537 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024464 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerName="manager" Sep 10 03:23:22.024703 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.024591 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" containerName="manager" Sep 10 03:23:22.028921 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.028898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.031981 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.031955 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 03:23:22.032105 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.031954 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 03:23:22.032105 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.031959 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 03:23:22.032105 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.031953 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lg6lz\"" Sep 10 03:23:22.037458 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.037386 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-xvg8l"] Sep 10 03:23:22.058847 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.058807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-metrics-certs\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.059057 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.058906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rcb\" (UniqueName: \"kubernetes.io/projected/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-kube-api-access-65rcb\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.059244 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.059219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-webhook-cert\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.159744 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.159698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-webhook-cert\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.159954 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.159766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-metrics-certs\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.159954 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.159803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rcb\" (UniqueName: \"kubernetes.io/projected/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-kube-api-access-65rcb\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.162329 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.162297 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-metrics-certs\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.162440 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.162329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-webhook-cert\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.167348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.167324 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rcb\" (UniqueName: \"kubernetes.io/projected/0fad42b1-0c8a-4d0f-9bb4-b739a028341c-kube-api-access-65rcb\") pod \"maas-controller-76d967fdff-xvg8l\" (UID: \"0fad42b1-0c8a-4d0f-9bb4-b739a028341c\") " pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.340849 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.340752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:22.613146 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:22.613066 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d967fdff-xvg8l"] Sep 10 03:23:22.616267 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:23:22.616235 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fad42b1_0c8a_4d0f_9bb4_b739a028341c.slice/crio-76e117eca2f86becdf49f4e711da962df7594abcb9c73cd5033893c563d9ad21 WatchSource:0}: Error finding container 76e117eca2f86becdf49f4e711da962df7594abcb9c73cd5033893c563d9ad21: Status 404 returned error can't find the container with id 76e117eca2f86becdf49f4e711da962df7594abcb9c73cd5033893c563d9ad21 Sep 10 03:23:23.001579 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:23.001549 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6072bcb-7b7b-4b7e-abc4-ee529727c7b0" path="/var/lib/kubelet/pods/d6072bcb-7b7b-4b7e-abc4-ee529727c7b0/volumes" Sep 10 03:23:23.263233 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:23.263144 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-xvg8l" event={"ID":"0fad42b1-0c8a-4d0f-9bb4-b739a028341c","Type":"ContainerStarted","Data":"7ea5f44b59d7b47cca88565e4a34cf5afa7ad87a343dfeb608564aa94bc2dbd7"} Sep 10 03:23:23.263233 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:23.263230 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d967fdff-xvg8l" event={"ID":"0fad42b1-0c8a-4d0f-9bb4-b739a028341c","Type":"ContainerStarted","Data":"76e117eca2f86becdf49f4e711da962df7594abcb9c73cd5033893c563d9ad21"} Sep 10 03:23:23.263428 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:23.263300 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:23:23.279389 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:23.279324 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76d967fdff-xvg8l" podStartSLOduration=0.815434164 podStartE2EDuration="1.279308212s" totalImagesPullingTime="463.874048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:23:23.278852691 +0000 UTC m=+2206.802709381" watchObservedRunningTime="2026-09-10 03:23:23.279308212 +0000 UTC m=+2206.803164900" Sep 10 03:23:34.270432 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:23:34.270397 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76d967fdff-xvg8l" Sep 10 03:26:37.184070 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:26:37.184039 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:26:37.199479 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:26:37.199454 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:27:27.143117 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.143079 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-2zb67"] Sep 10 03:27:27.156482 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.156241 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-2zb67"] Sep 10 03:27:27.156482 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.156269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.283280 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.283242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-openshift-service-ca\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.283463 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.283314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvt5\" (UniqueName: \"kubernetes.io/projected/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-kube-api-access-zkvt5\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.283463 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.283358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-maas-api-tls\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.283552 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.283463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-metrics-certs\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.384357 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.384318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-maas-api-tls\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.384542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.384374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-metrics-certs\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.384542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.384420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-openshift-service-ca\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.384542 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.384454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvt5\" (UniqueName: \"kubernetes.io/projected/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-kube-api-access-zkvt5\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.385173 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.385144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-openshift-service-ca\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.386944 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.386919 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-maas-api-tls\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.387079 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.386971 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-metrics-certs\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.391610 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.391585 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvt5\" (UniqueName: \"kubernetes.io/projected/39c3df2b-6f81-49d1-b7f4-dd2610b6e61b-kube-api-access-zkvt5\") pod \"maas-api-59f756556b-2zb67\" (UID: \"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.468502 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.468398 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:27.740824 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:27.740799 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-2zb67"] Sep 10 03:27:27.743528 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:27:27.743499 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c3df2b_6f81_49d1_b7f4_dd2610b6e61b.slice/crio-d9382e00885a47731f93c1f03d325b7975d67cfeee856fb4db7d58b5d0efaa2e WatchSource:0}: Error finding container d9382e00885a47731f93c1f03d325b7975d67cfeee856fb4db7d58b5d0efaa2e: Status 404 returned error can't find the container with id d9382e00885a47731f93c1f03d325b7975d67cfeee856fb4db7d58b5d0efaa2e Sep 10 03:27:28.469731 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:28.469640 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" event={"ID":"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b","Type":"ContainerStarted","Data":"f359b624944d3983d948dd1d70429910460f40e70cfdeeb5314679b0f06465e9"} Sep 10 03:27:28.469731 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:28.469694 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" event={"ID":"39c3df2b-6f81-49d1-b7f4-dd2610b6e61b","Type":"ContainerStarted","Data":"d9382e00885a47731f93c1f03d325b7975d67cfeee856fb4db7d58b5d0efaa2e"} Sep 10 03:27:28.470171 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:28.469771 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:28.487425 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:28.487371 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" podStartSLOduration=1.048443852 podStartE2EDuration="1.487352356s" totalImagesPullingTime="438.908504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:27:28.4857949 +0000 UTC m=+2452.009651587" watchObservedRunningTime="2026-09-10 03:27:28.487352356 +0000 UTC m=+2452.011209042" Sep 10 03:27:34.480130 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:34.480100 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f756556b-2zb67" Sep 10 03:27:42.133394 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:42.133351 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 03:27:42.133932 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:42.133588 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerName="maas-api" containerID="cri-o://7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b" gracePeriod=30 Sep 10 03:27:42.964612 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:42.964565 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.37:8443/health\": dial tcp 10.132.0.37:8443: connect: connection refused" Sep 10 03:27:43.385364 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.385337 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 03:27:43.435807 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.435773 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs\") pod \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " Sep 10 03:27:43.436006 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.435821 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq528\" (UniqueName: \"kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528\") pod \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " Sep 10 03:27:43.436006 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.435874 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca\") pod \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " Sep 10 03:27:43.436006 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.435904 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls\") pod \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\" (UID: \"93c7e412-b3c3-443c-8a43-2ca4fc067c21\") " Sep 10 03:27:43.436250 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.436218 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93c7e412-b3c3-443c-8a43-2ca4fc067c21" (UID: "93c7e412-b3c3-443c-8a43-2ca4fc067c21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 03:27:43.438060 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.438028 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93c7e412-b3c3-443c-8a43-2ca4fc067c21" (UID: "93c7e412-b3c3-443c-8a43-2ca4fc067c21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:27:43.438183 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.438062 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528" (OuterVolumeSpecName: "kube-api-access-gq528") pod "93c7e412-b3c3-443c-8a43-2ca4fc067c21" (UID: "93c7e412-b3c3-443c-8a43-2ca4fc067c21"). InnerVolumeSpecName "kube-api-access-gq528". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 03:27:43.438183 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.438089 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93c7e412-b3c3-443c-8a43-2ca4fc067c21" (UID: "93c7e412-b3c3-443c-8a43-2ca4fc067c21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 03:27:43.536828 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.536791 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq528\" (UniqueName: \"kubernetes.io/projected/93c7e412-b3c3-443c-8a43-2ca4fc067c21-kube-api-access-gq528\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:27:43.536828 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.536821 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c7e412-b3c3-443c-8a43-2ca4fc067c21-openshift-service-ca\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:27:43.536828 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.536830 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-maas-api-tls\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:27:43.536828 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.536839 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c7e412-b3c3-443c-8a43-2ca4fc067c21-metrics-certs\") on node \"ip-10-0-134-225.ec2.internal\" DevicePath \"\"" Sep 10 03:27:43.545131 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.545097 2583 generic.go:356] "Generic (PLEG): container finished" podID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerID="7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b" exitCode=0 Sep 10 03:27:43.545267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.545141 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" Sep 10 03:27:43.545267 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.545143 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" event={"ID":"93c7e412-b3c3-443c-8a43-2ca4fc067c21","Type":"ContainerDied","Data":"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b"} Sep 10 03:27:43.545348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.545275 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk" event={"ID":"93c7e412-b3c3-443c-8a43-2ca4fc067c21","Type":"ContainerDied","Data":"7e728925b938c4a215da4501aba2556fe1ac06a455810fb4366f6270c40e6f21"} Sep 10 03:27:43.545348 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.545305 2583 scope.go:117] "RemoveContainer" containerID="7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b" Sep 10 03:27:43.554728 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.554712 2583 scope.go:117] "RemoveContainer" containerID="7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b" Sep 10 03:27:43.555060 ip-10-0-134-225 kubenswrapper[2583]: E0910 03:27:43.555040 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b\": container with ID starting with 7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b not found: ID does not exist" containerID="7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b" Sep 10 03:27:43.555135 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.555073 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b"} err="failed to get container status \"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b\": rpc error: code = NotFound desc = could not find container \"7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b\": container with ID starting with 7e72c2343d0f6e3a7587b2349acd5e2c0932c676656baf185eaba26b87b5633b not found: ID does not exist" Sep 10 03:27:43.566694 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.566666 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 03:27:43.571619 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:43.571595 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-5d8kk"] Sep 10 03:27:44.999898 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:27:44.999845 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" path="/var/lib/kubelet/pods/93c7e412-b3c3-443c-8a43-2ca4fc067c21/volumes" Sep 10 03:28:06.851656 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:06.851626 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76d967fdff-xvg8l_0fad42b1-0c8a-4d0f-9bb4-b739a028341c/manager/0.log" Sep 10 03:28:07.325169 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:07.325139 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-cthp2_1d31c9d4-984c-4738-b117-1db458959ccd/copy-manifests/0.log" Sep 10 03:28:07.331123 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:07.331102 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-cthp2_1d31c9d4-984c-4738-b117-1db458959ccd/manager/0.log" Sep 10 03:28:07.798903 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:07.798874 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59f756556b-2zb67_39c3df2b-6f81-49d1-b7f4-dd2610b6e61b/maas-api/0.log" Sep 10 03:28:10.107676 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:10.107647 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf_f06d4ba4-1b0c-47f8-8dc2-08518fec0431/istio-proxy/0.log" Sep 10 03:28:10.225532 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:10.225501 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sw9nh_fed26a76-df1a-4017-97ef-a59cd4ccd31a/discovery/0.log" Sep 10 03:28:10.921086 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:10.921037 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff84dbc68-p77s6_3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46/router/0.log" Sep 10 03:28:11.259753 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:11.259718 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-pskkj_ae4ffc10-7160-486b-a54b-38775171179d/main/0.log" Sep 10 03:28:11.368377 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:11.368350 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-m88kx_73f8ff10-5cbc-45d5-ab5c-db44d31f023b/main/0.log" Sep 10 03:28:11.813889 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:11.813839 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cd92p_0bebe3a2-6dde-4e70-b6c0-5c3db6eb5fbc/main/0.log" Sep 10 03:28:20.935694 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:20.935663 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vbpc_307f0a5e-ff8f-47b1-9bce-c6f98ac1c4ff/global-pull-secret-syncer/0.log" Sep 10 03:28:21.094206 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:21.094173 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4m4vt_edeb603a-a5ab-4613-9b62-0a59baee7e49/konnectivity-agent/0.log" Sep 10 03:28:21.180787 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:21.180759 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-225.ec2.internal_e8c5ec7fd22f67956c3c0e7799a6de22/haproxy/0.log" Sep 10 03:28:27.788097 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:27.788023 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-vblts_848ff24f-562d-4a9b-be26-609c2b74a313/cluster-monitoring-operator/0.log" Sep 10 03:28:27.901909 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:27.901878 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6498c9dcbf-rvp4x_ba5cc5c6-7638-4f79-943d-0eeebf58e34e/metrics-server/0.log" Sep 10 03:28:27.940051 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:27.940020 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-t46f8_f8aca351-0593-4dee-bf0f-975c649e2153/monitoring-plugin/0.log" Sep 10 03:28:28.178037 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.177961 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvh29_9ed454f6-6806-4922-a351-f21016e9d5d5/node-exporter/0.log" Sep 10 03:28:28.203939 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.203911 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvh29_9ed454f6-6806-4922-a351-f21016e9d5d5/kube-rbac-proxy/0.log" Sep 10 03:28:28.227733 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.227711 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvh29_9ed454f6-6806-4922-a351-f21016e9d5d5/init-textfile/0.log" Sep 10 03:28:28.623562 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.623532 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-m2p7g_5f19ee98-0a8e-44db-bcc4-79395a654669/prometheus-operator/0.log" Sep 10 03:28:28.647033 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.647006 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-m2p7g_5f19ee98-0a8e-44db-bcc4-79395a654669/kube-rbac-proxy/0.log" Sep 10 03:28:28.731216 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.731189 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56c5cd9c7c-tncxz_a2a4c2f3-f2d1-490d-9894-31eaaf654ec7/telemeter-client/0.log" Sep 10 03:28:28.761887 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.761832 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56c5cd9c7c-tncxz_a2a4c2f3-f2d1-490d-9894-31eaaf654ec7/reload/0.log" Sep 10 03:28:28.788696 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:28.788671 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56c5cd9c7c-tncxz_a2a4c2f3-f2d1-490d-9894-31eaaf654ec7/kube-rbac-proxy/0.log" Sep 10 03:28:30.259266 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.259172 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt"] Sep 10 03:28:30.259668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.259550 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerName="maas-api" Sep 10 03:28:30.259668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.259561 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerName="maas-api" Sep 10 03:28:30.259668 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.259630 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c7e412-b3c3-443c-8a43-2ca4fc067c21" containerName="maas-api" Sep 10 03:28:30.262732 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.262712 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.265003 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.264984 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrsp9\"/\"default-dockercfg-6x58w\"" Sep 10 03:28:30.265646 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.265630 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"kube-root-ca.crt\"" Sep 10 03:28:30.265768 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.265747 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"openshift-service-ca.crt\"" Sep 10 03:28:30.287776 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.287736 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt"] Sep 10 03:28:30.373374 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.373340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-podres\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.373566 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.373390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-proc\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.373566 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.373456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvnl\" (UniqueName: \"kubernetes.io/projected/d5d87891-1bea-40d0-9b20-6d17723b94fb-kube-api-access-6xvnl\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.373566 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.373499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-lib-modules\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.373566 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.373517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-sys\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474168 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-proc\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474186 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvnl\" (UniqueName: \"kubernetes.io/projected/d5d87891-1bea-40d0-9b20-6d17723b94fb-kube-api-access-6xvnl\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-lib-modules\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-sys\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474266 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-proc\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474372 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474346 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-sys\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474389 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-lib-modules\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474418 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-podres\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.474580 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.474501 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5d87891-1bea-40d0-9b20-6d17723b94fb-podres\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.486443 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.486407 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvnl\" (UniqueName: \"kubernetes.io/projected/d5d87891-1bea-40d0-9b20-6d17723b94fb-kube-api-access-6xvnl\") pod \"perf-node-gather-daemonset-srxmt\" (UID: \"d5d87891-1bea-40d0-9b20-6d17723b94fb\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.572336 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.572224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:30.861227 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:30.861056 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt"] Sep 10 03:28:30.864721 ip-10-0-134-225 kubenswrapper[2583]: W0910 03:28:30.864693 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd5d87891_1bea_40d0_9b20_6d17723b94fb.slice/crio-6a49d3b427925567143363050c767a2af3ca738b54b949c68c74acebda993b10 WatchSource:0}: Error finding container 6a49d3b427925567143363050c767a2af3ca738b54b949c68c74acebda993b10: Status 404 returned error can't find the container with id 6a49d3b427925567143363050c767a2af3ca738b54b949c68c74acebda993b10 Sep 10 03:28:31.133240 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.133157 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/2.log" Sep 10 03:28:31.136682 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.136656 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vtrdr_be0e6efc-131e-4866-851c-27b681b19cfd/console-operator/3.log" Sep 10 03:28:31.721225 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.721193 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84ccbdcc64-w4cgx_652f8d84-c710-4ff6-9e64-acd63a03da75/console/0.log" Sep 10 03:28:31.765046 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.765015 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-6mqnf_80363e78-04f3-4944-b63f-83ff136f2ed9/download-server/0.log" Sep 10 03:28:31.781727 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.781693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" event={"ID":"d5d87891-1bea-40d0-9b20-6d17723b94fb","Type":"ContainerStarted","Data":"eef4d115e5b0616e9f2483666783fea1d684ea22de554028715b52b12f305b6c"} Sep 10 03:28:31.781926 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.781752 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" event={"ID":"d5d87891-1bea-40d0-9b20-6d17723b94fb","Type":"ContainerStarted","Data":"6a49d3b427925567143363050c767a2af3ca738b54b949c68c74acebda993b10"} Sep 10 03:28:31.781926 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.781906 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:31.802680 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:31.802628 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" podStartSLOduration=1.802610203 podStartE2EDuration="1.802610203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 03:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:28:31.802072366 +0000 UTC m=+2515.325929053" watchObservedRunningTime="2026-09-10 03:28:31.802610203 +0000 UTC m=+2515.326466928" Sep 10 03:28:32.432456 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:32.432424 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-tp8vp_a5acc330-46a8-4ae6-91e3-1e1a89503ecd/volume-data-source-validator/0.log" Sep 10 03:28:33.603771 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:33.603738 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pw56k_0388e7f9-1f22-4244-9dfb-e6cf87020d8c/dns/0.log" Sep 10 03:28:33.637430 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:33.637391 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pw56k_0388e7f9-1f22-4244-9dfb-e6cf87020d8c/kube-rbac-proxy/0.log" Sep 10 03:28:33.795302 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:33.795277 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp6n2_9f135ece-4051-422d-a794-e4ac35d8c8b7/dns-node-resolver/0.log" Sep 10 03:28:34.404262 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:34.404232 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zx4l6_0ab99a0f-f0bc-4500-b53e-5b10a6cfba07/node-ca/0.log" Sep 10 03:28:35.318093 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:35.318062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf_f06d4ba4-1b0c-47f8-8dc2-08518fec0431/istio-proxy/0.log" Sep 10 03:28:35.604095 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:35.604010 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sw9nh_fed26a76-df1a-4017-97ef-a59cd4ccd31a/discovery/0.log" Sep 10 03:28:35.795305 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:35.795277 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff84dbc68-p77s6_3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46/router/0.log" Sep 10 03:28:36.700032 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:36.700004 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mdtx_1cdf66b1-4727-4e1c-977b-448c12a39d00/serve-healthcheck-canary/0.log" Sep 10 03:28:37.232772 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.232743 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kj7w5_419faac4-860c-472c-b9c7-1fb983cc3085/insights-operator/0.log" Sep 10 03:28:37.233441 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.233421 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kj7w5_419faac4-860c-472c-b9c7-1fb983cc3085/insights-operator/1.log" Sep 10 03:28:37.254036 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.254008 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77jcb_1a116314-9fb9-4786-b01a-1bbb80aef26a/kube-rbac-proxy/0.log" Sep 10 03:28:37.280677 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.280653 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77jcb_1a116314-9fb9-4786-b01a-1bbb80aef26a/exporter/0.log" Sep 10 03:28:37.304102 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.304074 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77jcb_1a116314-9fb9-4786-b01a-1bbb80aef26a/extractor/0.log" Sep 10 03:28:37.796476 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:37.796446 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-srxmt" Sep 10 03:28:40.615721 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:40.615685 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76d967fdff-xvg8l_0fad42b1-0c8a-4d0f-9bb4-b739a028341c/manager/0.log" Sep 10 03:28:40.816044 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:40.816011 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf_f06d4ba4-1b0c-47f8-8dc2-08518fec0431/istio-proxy/0.log" Sep 10 03:28:40.898919 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:40.898809 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-cthp2_1d31c9d4-984c-4738-b117-1db458959ccd/manager/0.log" Sep 10 03:28:40.921409 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:40.921379 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-cthp2_1d31c9d4-984c-4738-b117-1db458959ccd/copy-manifests/0.log" Sep 10 03:28:41.111158 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:41.111128 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sw9nh_fed26a76-df1a-4017-97ef-a59cd4ccd31a/discovery/0.log" Sep 10 03:28:41.252895 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:41.252868 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff84dbc68-p77s6_3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46/router/0.log" Sep 10 03:28:42.319541 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:42.319508 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cd97c998c-q7b6w_c102fa5d-ba57-4644-bcde-a9e36c59a3bb/manager/0.log" Sep 10 03:28:42.719814 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:42.719734 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4xnsf_f06d4ba4-1b0c-47f8-8dc2-08518fec0431/istio-proxy/0.log" Sep 10 03:28:43.025435 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:43.025399 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sw9nh_fed26a76-df1a-4017-97ef-a59cd4ccd31a/discovery/0.log" Sep 10 03:28:43.199803 ip-10-0-134-225 kubenswrapper[2583]: I0910 03:28:43.199771 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff84dbc68-p77s6_3f8e36f0-f1d8-40fa-a5c6-3da94dc53e46/router/0.log"