Sep 15 21:15:59.261093 ip-10-0-133-61 systemd[1]: Starting Kubernetes Kubelet... Sep 15 21:15:59.610598 ip-10-0-133-61 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:59.610598 ip-10-0-133-61 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 21:15:59.610598 ip-10-0-133-61 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:59.610598 ip-10-0-133-61 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 21:15:59.610598 ip-10-0-133-61 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:59.612803 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.612708 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 21:15:59.617208 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617192 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.617208 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617207 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617212 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617215 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617218 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617221 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617225 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617227 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617230 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617233 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617236 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617239 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617242 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617245 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617248 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617255 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617258 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617261 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617263 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617266 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617269 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.617268 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617272 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617275 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617278 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617281 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617284 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617287 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617290 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617292 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617295 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617297 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617302 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617306 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617308 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617312 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617315 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617319 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617321 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617324 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617327 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617329 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.617747 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617332 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617335 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617338 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617340 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617343 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617345 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617348 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617350 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617353 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617355 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617358 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617360 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617363 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617366 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617370 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617373 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617376 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617379 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617382 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617384 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.618249 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617387 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617390 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617392 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617394 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617397 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617399 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617402 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617405 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617407 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617410 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617413 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617415 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617418 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617423 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617427 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617430 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617432 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617435 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617438 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.618727 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617441 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617445 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617448 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617451 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617454 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617457 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617460 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617463 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617466 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617469 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617472 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617474 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617477 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617480 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617482 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617485 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617487 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617490 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.617493 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618857 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.619198 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618866 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618869 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618872 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618875 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618880 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618883 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618886 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618888 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618891 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618894 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618897 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618899 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618902 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618905 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618907 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618910 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618913 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618916 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618919 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618922 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.619794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618924 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618927 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618930 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618933 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618935 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618938 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618940 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618943 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618945 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618948 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618950 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618953 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618956 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618958 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618960 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618963 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618966 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618969 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618971 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618974 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.620324 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618976 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618979 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618982 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618984 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618986 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618989 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618991 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618994 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618997 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.618999 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619006 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619011 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619014 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619017 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619019 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619022 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619025 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619027 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619030 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619032 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.620813 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619035 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619038 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619041 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619043 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619046 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619048 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619051 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619054 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619056 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619059 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619062 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619064 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619067 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619069 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619072 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619074 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619077 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619079 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619082 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619084 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.621325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619088 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619090 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619095 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619097 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619100 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619102 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619106 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619110 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619113 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619116 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619118 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619121 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619124 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619127 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619131 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619134 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619137 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619139 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619517 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.621808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619522 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619525 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619528 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619533 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619537 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619539 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619543 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619545 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619548 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619551 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619555 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619557 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619560 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619563 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619565 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619569 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619572 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619575 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619577 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619580 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.622293 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619583 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619585 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619588 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619591 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619593 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619596 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619598 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619601 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619603 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619606 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619608 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619611 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619613 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619616 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619618 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619621 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619624 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619627 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619629 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619631 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.622780 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619634 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619638 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619642 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619645 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619647 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619650 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619653 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619655 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619658 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619661 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619663 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619666 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619668 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619671 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619673 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619676 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619678 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619681 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619684 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.623300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619686 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619689 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619691 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619694 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619696 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619699 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619702 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619704 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619707 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619710 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619712 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619715 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619717 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619720 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619722 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619725 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619727 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619729 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619732 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619735 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.623839 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619737 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619740 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619743 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619748 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619752 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619754 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619757 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619760 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619763 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619765 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619768 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619770 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619773 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619776 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619780 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619783 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619785 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619788 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.619791 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.624392 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619876 2552 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619910 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619915 2552 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619920 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619925 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619928 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619932 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619937 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619940 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619943 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619946 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619949 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619952 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619955 2552 flags.go:64] FLAG: --cgroup-root="" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619959 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619962 2552 flags.go:64] FLAG: --client-ca-file="" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619965 2552 flags.go:64] FLAG: --cloud-provider="external" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619968 2552 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619972 2552 flags.go:64] FLAG: --cluster-domain="" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619974 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619978 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619981 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619985 2552 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619989 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 21:15:59.624950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619992 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619995 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.619998 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620001 2552 flags.go:64] FLAG: --contention-profiling="false" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620004 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620008 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620011 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620014 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620018 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620021 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620023 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620026 2552 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620031 2552 flags.go:64] FLAG: --enable-server="true" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620035 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620039 2552 flags.go:64] FLAG: --event-burst="100" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620042 2552 flags.go:64] FLAG: --event-qps="50" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620045 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620048 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620051 2552 flags.go:64] FLAG: --eviction-hard="" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620055 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620057 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620060 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620064 2552 flags.go:64] FLAG: --eviction-soft="" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620067 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620070 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 21:15:59.625554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620073 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620076 2552 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620079 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620082 2552 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620085 2552 flags.go:64] FLAG: --feature-gates="" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620091 2552 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620095 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620098 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620101 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620104 2552 flags.go:64] FLAG: --healthz-port="10248" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620107 2552 flags.go:64] FLAG: --help="false" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620110 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620114 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620117 2552 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620120 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620123 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620126 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620129 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620132 2552 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620136 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620139 2552 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620495 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620499 2552 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620501 2552 flags.go:64] FLAG: --kube-reserved="" Sep 15 21:15:59.626197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620505 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620507 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620511 2552 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620513 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620516 2552 flags.go:64] FLAG: --lock-file="" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620520 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620522 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620525 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620531 2552 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620533 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620536 2552 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620539 2552 flags.go:64] FLAG: --logging-format="text" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620542 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620545 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620548 2552 flags.go:64] FLAG: --manifest-url="" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620551 2552 flags.go:64] FLAG: --manifest-url-header="" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620555 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620558 2552 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620562 2552 flags.go:64] FLAG: --max-pods="110" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620565 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620568 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620571 2552 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620574 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620577 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620580 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 21:15:59.626792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620582 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620590 2552 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620594 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620597 2552 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620599 2552 flags.go:64] FLAG: --pod-cidr="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620603 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620609 2552 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620612 2552 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620615 2552 flags.go:64] FLAG: --pods-per-core="0" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620618 2552 flags.go:64] FLAG: --port="10250" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620621 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620624 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-021295312c90e14fb" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620628 2552 flags.go:64] FLAG: --qos-reserved="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620634 2552 flags.go:64] FLAG: --read-only-port="10255" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620637 2552 flags.go:64] FLAG: --register-node="true" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620640 2552 flags.go:64] FLAG: --register-with-taints="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620644 2552 flags.go:64] FLAG: --registry-burst="10" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620647 2552 flags.go:64] FLAG: --registry-qps="5" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620650 2552 flags.go:64] FLAG: --reserved-cpus="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620652 2552 flags.go:64] FLAG: --reserved-memory="" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620656 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620659 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620662 2552 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620665 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620668 2552 flags.go:64] FLAG: --runonce="false" Sep 15 21:15:59.627404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620671 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620674 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620677 2552 flags.go:64] FLAG: --seccomp-default="false" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620680 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620683 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620685 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620689 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620692 2552 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620694 2552 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620699 2552 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620702 2552 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620705 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620708 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620711 2552 flags.go:64] FLAG: --system-cgroups="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620714 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620720 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620722 2552 flags.go:64] FLAG: --tls-cert-file="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620725 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620729 2552 flags.go:64] FLAG: --tls-min-version="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620732 2552 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620736 2552 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620739 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620742 2552 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620745 2552 flags.go:64] FLAG: --v="2" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620748 2552 flags.go:64] FLAG: --version="false" Sep 15 21:15:59.628042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620753 2552 flags.go:64] FLAG: --vmodule="" Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620756 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.620759 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620946 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620961 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620965 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620968 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620972 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620975 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620978 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620981 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620984 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620987 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620990 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620993 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620996 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.620998 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621001 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621004 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621007 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.628675 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621009 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621012 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621015 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621018 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621020 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621023 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621026 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621030 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621034 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621037 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621040 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621042 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621045 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621048 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621050 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621054 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621060 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621063 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621067 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.629364 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621070 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621073 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621075 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621078 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621081 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621083 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621086 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621088 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621091 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621094 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621096 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621099 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621102 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621105 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621108 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621111 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621113 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621116 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621119 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621121 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.630071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621124 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621127 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621129 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621132 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621135 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621137 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621140 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621142 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621146 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621151 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621155 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621158 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621160 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621163 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621166 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621169 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621171 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621174 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621176 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621179 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.630581 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621181 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621184 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621186 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621190 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621192 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621195 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621197 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621200 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621203 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621205 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621208 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621210 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621213 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621216 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621219 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621222 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621224 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621227 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621230 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621232 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.631136 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621235 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621238 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.621240 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.621667 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.628667 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.628683 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.628703 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.628708 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628962 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628974 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628980 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628986 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628991 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.628995 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629006 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.631617 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629011 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629016 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629020 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629024 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629029 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629033 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629038 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629042 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629046 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629051 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629056 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629065 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629070 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629077 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629084 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629088 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629094 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629099 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629103 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.632083 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629110 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629117 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629123 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629128 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629134 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629144 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629149 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629153 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629157 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629162 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629166 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629170 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629174 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629178 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629183 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629187 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629192 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629201 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629208 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.632574 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629214 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629220 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629226 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629230 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629236 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629240 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629245 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629249 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629254 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629259 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629268 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629272 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629276 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629281 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629285 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629290 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629294 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629298 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629303 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629307 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.633106 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629311 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629316 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629324 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629329 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629333 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629338 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629342 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629346 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629350 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629355 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629358 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629363 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629367 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629372 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629376 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629385 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629390 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629394 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629398 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629403 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.633588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629407 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629412 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629416 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629420 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629424 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629428 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629432 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629436 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629445 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629449 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629456 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629462 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629466 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.629471 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.634102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.629479 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630093 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630106 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630109 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630112 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630116 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630119 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630123 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630125 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630128 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630131 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630134 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630137 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630139 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630142 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630145 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630147 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630150 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630153 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:59.634445 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630156 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630161 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630165 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630168 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630171 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630174 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630177 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630180 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630184 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630187 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630189 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630192 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630195 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630197 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630200 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630202 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630205 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630208 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630210 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630213 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:59.634924 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630215 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630218 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630221 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630223 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630226 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630228 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630231 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630233 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630236 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630240 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630243 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630246 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630249 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630252 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630255 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630258 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630261 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630263 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630266 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630269 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:59.635410 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630271 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630274 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630276 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630281 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630284 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630287 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630289 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630292 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630294 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630297 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630300 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630302 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630305 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630307 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630310 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630313 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630315 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630318 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630320 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:59.636013 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630323 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630326 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630328 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630331 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630334 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630338 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630340 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630343 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630346 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630348 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630351 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630353 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630356 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630359 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630361 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630364 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630366 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630369 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630371 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630374 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:59.636465 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630376 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:15:59.630379 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.630384 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.630500 2552 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.632729 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.633632 2552 server.go:1019] "Starting client certificate rotation" Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.633726 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.633760 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:15:59.636985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.636076 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:15:59.637248 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.637230 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:15:59.645516 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.645500 2552 log.go:25] "Validated CRI v1 runtime API" Sep 15 21:15:59.650391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.650377 2552 log.go:25] "Validated CRI v1 image API" Sep 15 21:15:59.651275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.651261 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 21:15:59.654931 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.654911 2552 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 afa5318d-6666-4ec8-a005-061f735c0ea4:/dev/nvme0n1p3 ee9c36cb-7b88-4431-a934-b92c3dc1d61c:/dev/nvme0n1p4] Sep 15 21:15:59.655004 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.654930 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 21:15:59.659779 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.659760 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:15:59.660971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.660850 2552 manager.go:216] Machine: {Timestamp:2026-09-15 21:15:59.659625417 +0000 UTC m=+0.294777519 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098752 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9a5d163473bbb4d9e90313d190c4 SystemUUID:ec2f9a5d-1634-73bb-b4d9-e90313d190c4 BootID:16c54309-37b4-4b25-9f65-c50ea3ca45a4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:43:06:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:43:06:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:ad:4d:73:ff:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 21:15:59.660971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.660970 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 21:15:59.661127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.661070 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 21:15:59.664159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.664138 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 21:15:59.664328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.664161 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 21:15:59.664975 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.664959 2552 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 21:15:59.665039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.664981 2552 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 21:15:59.665039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.664992 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:15:59.665638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.665627 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:15:59.666736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.666725 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 21:15:59.671257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671244 2552 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:15:59.671399 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671387 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 21:15:59.671457 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671428 2552 kubelet.go:486] "Attempting to sync node with API server" Sep 15 21:15:59.671457 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671450 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 21:15:59.671546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671471 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 21:15:59.671546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671485 2552 kubelet.go:398] "Adding apiserver pod source" Sep 15 21:15:59.671546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671516 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 21:15:59.671967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671953 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:15:59.672029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.671975 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:15:59.675453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.675406 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 21:15:59.678149 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.678131 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 21:15:59.678224 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.678164 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 21:15:59.680188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680172 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 21:15:59.680188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680189 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680196 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680201 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680210 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680215 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680229 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680233 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680240 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680248 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680262 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 21:15:59.680317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680267 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 21:15:59.680971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680961 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 21:15:59.680971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.680972 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 21:15:59.681045 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.681029 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 21:15:59.681192 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.681172 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 21:15:59.684339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.684320 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 21:15:59.684665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.684653 2552 server.go:1268] "Started kubelet" Sep 15 21:15:59.684741 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.684709 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 21:15:59.684815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.684767 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 21:15:59.684979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.684957 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 21:15:59.685492 ip-10-0-133-61 systemd[1]: Started Kubernetes Kubelet. Sep 15 21:15:59.686329 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.686313 2552 server.go:310] "Adding debug handlers to kubelet server" Sep 15 21:15:59.686461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.686447 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 21:15:59.688084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.687208 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qcg9" Sep 15 21:15:59.689636 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.688581 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-61.ec2.internal.18d59b193f1766d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-61.ec2.internal,UID:ip-10-0-133-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-61.ec2.internal,},FirstTimestamp:2026-09-15 21:15:59.684626132 +0000 UTC m=+0.319778234,LastTimestamp:2026-09-15 21:15:59.684626132 +0000 UTC m=+0.319778234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-61.ec2.internal,}" Sep 15 21:15:59.690607 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.690585 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 21:15:59.692854 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.692836 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 21:15:59.693180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.693163 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qcg9" Sep 15 21:15:59.693390 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.693371 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 21:15:59.693570 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.693534 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:15:59.694124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694105 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 21:15:59.694124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694106 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 21:15:59.694279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694136 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 21:15:59.694279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694227 2552 reconstruct.go:97] "Volume reconstruction finished" Sep 15 21:15:59.694279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694237 2552 reconciler.go:29] "Reconciler: start to sync state" Sep 15 21:15:59.694884 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694868 2552 factory.go:55] Registering systemd factory Sep 15 21:15:59.694957 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.694901 2552 factory.go:223] Registration of the systemd container factory successfully Sep 15 21:15:59.695105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695092 2552 factory.go:153] Registering CRI-O factory Sep 15 21:15:59.695105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695106 2552 factory.go:223] Registration of the crio container factory successfully Sep 15 21:15:59.695194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695146 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 21:15:59.695194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695164 2552 factory.go:103] Registering Raw factory Sep 15 21:15:59.695194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695176 2552 manager.go:1196] Started watching for new ooms in manager Sep 15 21:15:59.695572 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.695556 2552 manager.go:349] Starting recovery of all containers Sep 15 21:15:59.705672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.705650 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:59.707875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.707857 2552 manager.go:354] Recovery completed Sep 15 21:15:59.708539 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.708518 2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.712641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.712627 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.714354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714340 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.714424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714365 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.714424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714375 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.714993 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714971 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 21:15:59.714993 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714983 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.715106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.714982 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 21:15:59.715106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.715045 2552 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:15:59.715658 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.715642 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.715735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.715668 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.715735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.715678 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.716157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.716145 2552 policy_none.go:49] "None policy: Start" Sep 15 21:15:59.716195 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.716163 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 21:15:59.716195 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.716173 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 21:15:59.717270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.717255 2552 policy_none.go:47] "Start" Sep 15 21:15:59.757617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.757478 2552 manager.go:337] "Starting Device Plugin manager" Sep 15 21:15:59.757617 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.757645 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 21:15:59.757617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.757658 2552 server.go:87] "Starting device plugin registration server" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.757975 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.757986 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.758089 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.758175 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.758185 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.758655 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 21:15:59.761731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.758696 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:15:59.782247 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.782222 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 21:15:59.783505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.783487 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 21:15:59.783598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.783533 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 21:15:59.783598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.783553 2552 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 21:15:59.783697 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.783632 2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 21:15:59.786025 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.786005 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:59.858288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.858274 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.859190 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.859173 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.859258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.859206 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.859258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.859217 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.859258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.859247 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.867511 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.867455 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.867511 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.867476 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-61.ec2.internal\": node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:15:59.884365 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.884346 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal"] Sep 15 21:15:59.884445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.884419 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.885228 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.885212 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.885317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.885244 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.885317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.885257 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.887408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.887394 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.887558 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.887541 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.887615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.887598 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.888095 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888075 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.888196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888107 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.888196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888118 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.888196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888077 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.888196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888137 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.888196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.888147 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.888505 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.888492 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:15:59.890423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.890409 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.890506 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.890441 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:59.892217 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.892186 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:59.892217 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.892211 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:59.892355 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.892228 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:59.895699 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.895683 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9572ef115be1ab8679034b1a50b59a2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"f9572ef115be1ab8679034b1a50b59a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.895776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.895706 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.895776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.895726 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.914441 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.914425 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.918920 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.918904 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.988945 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:15:59.988919 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:15:59.995967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.995940 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.995967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.995968 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.996153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.995986 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9572ef115be1ab8679034b1a50b59a2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"f9572ef115be1ab8679034b1a50b59a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.996153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.996038 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.996153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.996054 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9572ef115be1ab8679034b1a50b59a2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"f9572ef115be1ab8679034b1a50b59a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:15:59.996153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:15:59.996065 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8484a1774e051d9e3b56ffda320809a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"e8484a1774e051d9e3b56ffda320809a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:16:00.089713 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.089687 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.190453 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.190391 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.217834 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.217813 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:16:00.223128 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.223113 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:16:00.291357 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.291325 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.391858 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.391814 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.493839 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.493789 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.595367 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.595320 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.634731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.634698 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 21:16:00.635300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.634889 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 21:16:00.635300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.634891 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 21:16:00.695769 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.695749 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.698069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.698032 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:10:59 +0000 UTC" deadline="2028-05-05 14:36:00.04262198 +0000 UTC" Sep 15 21:16:00.698069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.698068 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14345h19m59.344557472s" Sep 15 21:16:00.796813 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.796781 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.801661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:00.801643 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:16:00.897463 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.897416 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:00.999932 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:00.999903 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:01.100444 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.100366 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:01.184234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.184209 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:16:01.201508 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.201488 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Sep 15 21:16:01.247318 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.247292 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:16:01.296509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.296479 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Sep 15 21:16:01.309910 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.309889 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:16:01.310676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.310663 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Sep 15 21:16:01.320103 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.320082 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:16:01.603874 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.603843 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:16:01.651294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.651264 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:16:01.673006 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.672979 2552 apiserver.go:52] "Watching apiserver" Sep 15 21:16:01.680183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.680155 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 21:16:01.681728 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.681702 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6w6wz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","openshift-network-diagnostics/network-check-target-m2l6s","openshift-network-operator/iptables-alerter-z5gfr","openshift-ovn-kubernetes/ovnkube-node-qrpkm","kube-system/konnectivity-agent-vf44d","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2kwrn","openshift-dns/node-resolver-fx9kz","openshift-image-registry/node-ca-hj8x9","openshift-multus/multus-5p8jk","openshift-multus/multus-additional-cni-plugins-p7r2c","openshift-multus/network-metrics-daemon-9fzbx"] Sep 15 21:16:01.683410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.683386 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.684842 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.684807 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.685939 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.685920 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 21:16:01.686055 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686036 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 21:16:01.686115 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.685924 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.686269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686251 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.686530 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686355 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.686530 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686398 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnkn6\"" Sep 15 21:16:01.686723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686696 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 21:16:01.686896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.686742 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687308 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687362 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687400 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687423 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687803 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hfzd\"" Sep 15 21:16:01.687983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.687856 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 21:16:01.688309 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.688285 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.688379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.688330 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.688693 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.688670 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.688766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.688712 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 21:16:01.688950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.688933 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lptwv\"" Sep 15 21:16:01.689450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.689431 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.689537 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.689498 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6zsv\"" Sep 15 21:16:01.689537 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.689499 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.690252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.690230 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.691282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.691024 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 21:16:01.691282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.691095 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 21:16:01.691282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.691113 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2h9vl\"" Sep 15 21:16:01.691579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.691564 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.691656 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.691635 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:01.692705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.692685 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhdv7\"" Sep 15 21:16:01.692924 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.692908 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.692983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.692972 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.693115 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.693099 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 21:16:01.693640 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.693629 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 21:16:01.694581 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.694564 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:01.694671 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.694644 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:01.696077 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.696059 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.697703 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.697687 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.698148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.698131 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.698397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.698378 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-czth7\"" Sep 15 21:16:01.698484 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.698441 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.699448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.699429 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.699584 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.699555 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 21:16:01.700257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.699945 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 21:16:01.700257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.700047 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 21:16:01.700257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.700054 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8vzmv\"" Sep 15 21:16:01.700921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.700900 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.701025 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.700969 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:01.702628 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.702187 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 21:16:01.702628 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.702282 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 21:16:01.702628 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.702586 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbrgb\"" Sep 15 21:16:01.703143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703120 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703644 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-bin\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d68\" (UniqueName: \"kubernetes.io/projected/de262f3f-e229-4776-8132-37278c6f7695-kube-api-access-r4d68\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703700 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-cnibin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703720 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-multus-daemon-config\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703740 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-kubernetes\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703759 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de262f3f-e229-4776-8132-37278c6f7695-host-slash\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-hostroot\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703795 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-etc-kubernetes\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703815 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvfl\" (UniqueName: \"kubernetes.io/projected/3175aee1-79d4-448c-93ea-03d67123e57d-kube-api-access-7hvfl\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703857 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.703886 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-log-socket\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704175 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-tmp\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704242 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-k8s-cni-cncf-io\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.704278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704274 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704305 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de262f3f-e229-4776-8132-37278c6f7695-iptables-alerter-script\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704391 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovn-node-metrics-cert\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704424 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-systemd\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704449 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-system-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704495 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcjw\" (UniqueName: \"kubernetes.io/projected/e4c739ad-96b5-47dd-8dfa-374af65e3165-kube-api-access-jtcjw\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704528 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f3178ee-7869-410c-9852-308038c9bf62-agent-certs\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704559 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-multus\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704586 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-slash\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704601 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-netns\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704652 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-node-log\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704702 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-socket-dir-parent\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704755 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsm7f\" (UniqueName: \"kubernetes.io/projected/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-kube-api-access-bsm7f\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704802 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-conf\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704857 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-host\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704885 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-kubelet\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704916 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-lib-modules\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704946 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-tuned\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.704977 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-cni-binary-copy\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705009 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-bin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705051 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-conf-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705079 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-serviceca\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705109 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-systemd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705141 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-script-lib\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705174 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f3178ee-7869-410c-9852-308038c9bf62-konnectivity-ca\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705200 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-etc-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705231 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-env-overrides\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705275 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlmg\" (UniqueName: \"kubernetes.io/projected/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-kube-api-access-7zlmg\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705321 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysconfig\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705351 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-var-lib-kubelet\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705380 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.705762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705405 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-kubelet\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705451 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-dbus\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705510 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705536 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-var-lib-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705577 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-netd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705732 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-kubelet-config\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-ovn\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705808 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-run\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705849 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-config\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-multus-certs\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705913 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-modprobe-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705942 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-sys\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705971 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-host\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.705995 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-netns\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.706488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.706025 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-systemd-units\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.727031 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.727001 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g8ckd" Sep 15 21:16:01.742588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.742567 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g8ckd" Sep 15 21:16:01.788522 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.788474 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"f9572ef115be1ab8679034b1a50b59a2","Type":"ContainerStarted","Data":"6447b00e76f64864565aa0948ca84f8338dddc7d4a4a3bbb5396094dbc04873a"} Sep 15 21:16:01.795114 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.795094 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 21:16:01.806822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806797 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.806954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806847 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-log-socket\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.806954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806873 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.806954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-tmp\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.806954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806920 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-k8s-cni-cncf-io\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.806954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806944 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.806960 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806971 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806997 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-k8s-cni-cncf-io\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.806967 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807026 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-log-socket\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.807037 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:02.307013392 +0000 UTC m=+2.942165488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807051 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807074 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de262f3f-e229-4776-8132-37278c6f7695-iptables-alerter-script\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807108 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807109 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.807172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807135 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovn-node-metrics-cert\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807205 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-systemd\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807233 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-system-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcjw\" (UniqueName: \"kubernetes.io/projected/e4c739ad-96b5-47dd-8dfa-374af65e3165-kube-api-access-jtcjw\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807284 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f3178ee-7869-410c-9852-308038c9bf62-agent-certs\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807310 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807324 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807333 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807359 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807386 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-multus\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807391 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-system-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807407 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-slash\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807428 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-netns\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807449 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807451 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-multus\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807470 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-node-log\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807491 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sll2w\" (UniqueName: \"kubernetes.io/projected/757e5f7e-9388-48af-bc43-6ed61d49a316-kube-api-access-sll2w\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.807691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807516 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807536 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-socket-dir-parent\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807556 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsm7f\" (UniqueName: \"kubernetes.io/projected/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-kube-api-access-bsm7f\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807577 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-conf\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807603 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757e5f7e-9388-48af-bc43-6ed61d49a316-tmp-dir\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807627 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-host\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807333 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-systemd\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807659 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de262f3f-e229-4776-8132-37278c6f7695-iptables-alerter-script\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807683 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807661 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-host\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807667 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807699 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-slash\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807703 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-kubelet\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807726 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-kubelet\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807740 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-node-log\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807736 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-lib-modules\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807740 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-run-netns\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807778 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-tuned\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.808504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807795 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-socket-dir-parent\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807856 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysctl-conf\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807863 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-system-cni-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807894 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-cni-binary-copy\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807921 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-bin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807945 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-lib-modules\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807967 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-conf-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.807992 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-serviceca\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808016 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-systemd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808038 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-registration-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808073 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808121 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-script-lib\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808147 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f3178ee-7869-410c-9852-308038c9bf62-konnectivity-ca\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808168 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757e5f7e-9388-48af-bc43-6ed61d49a316-hosts-file\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-cnibin\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808216 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-etc-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808237 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-env-overrides\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808256 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlmg\" (UniqueName: \"kubernetes.io/projected/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-kube-api-access-7zlmg\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808277 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hw8z\" (UniqueName: \"kubernetes.io/projected/e70176b6-167d-40d3-81ff-d27f61b9ad08-kube-api-access-7hw8z\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808298 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysconfig\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808373 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-var-lib-kubelet\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808394 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808417 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-kubelet\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808440 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-dbus\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808460 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808480 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-var-lib-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808501 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-netd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808521 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-kubelet-config\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808542 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-ovn\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808564 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-run\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808583 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-cni-binary-copy\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808598 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-config\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808621 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-device-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808632 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-cni-bin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.809966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808644 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dt69\" (UniqueName: \"kubernetes.io/projected/7794745f-2df5-4a52-8ce7-e2d4627909c2-kube-api-access-7dt69\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808687 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-sysconfig\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808709 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-cni-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808724 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-var-lib-kubelet\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808755 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-multus-conf-dir\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808771 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-dbus\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808787 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-var-lib-kubelet\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808840 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-var-lib-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808881 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-netd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808927 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0487501f-1be4-4d51-8ba4-8314af778987-kubelet-config\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.808990 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-ovn\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809167 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-run-systemd\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809474 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-serviceca\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809528 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-config\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809536 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-run\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809687 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovnkube-script-lib\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.809741 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-etc-openvswitch\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810049 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f3178ee-7869-410c-9852-308038c9bf62-konnectivity-ca\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.810766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810110 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-multus-certs\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810157 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-modprobe-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810183 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-sys\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810227 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-host\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-socket-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810305 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-netns\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810319 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-env-overrides\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-systemd-units\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-bin\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810427 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d68\" (UniqueName: \"kubernetes.io/projected/de262f3f-e229-4776-8132-37278c6f7695-kube-api-access-r4d68\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810476 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-cnibin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810513 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-multus-daemon-config\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810543 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-sys-fs\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810571 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v5c\" (UniqueName: \"kubernetes.io/projected/3187d9d1-bd94-45fd-a140-28094411a7c0-kube-api-access-n6v5c\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-kubernetes\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810685 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de262f3f-e229-4776-8132-37278c6f7695-host-slash\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810712 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-hostroot\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.811645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810737 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-kubernetes\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810739 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-etc-kubernetes\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810775 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-etc-kubernetes\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810787 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvfl\" (UniqueName: \"kubernetes.io/projected/3175aee1-79d4-448c-93ea-03d67123e57d-kube-api-access-7hvfl\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.810818 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-cnibin\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-systemd-units\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811251 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3175aee1-79d4-448c-93ea-03d67123e57d-multus-daemon-config\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811626 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-ovn-node-metrics-cert\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811688 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-netns\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811272 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-sys\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811722 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-host-run-multus-certs\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811767 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-modprobe-d\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811798 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f3178ee-7869-410c-9852-308038c9bf62-agent-certs\") pod \"konnectivity-agent-vf44d\" (UID: \"6f3178ee-7869-410c-9852-308038c9bf62\") " pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811814 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de262f3f-e229-4776-8132-37278c6f7695-host-slash\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811788 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4c739ad-96b5-47dd-8dfa-374af65e3165-host\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811771 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3175aee1-79d4-448c-93ea-03d67123e57d-hostroot\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811771 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-host-cni-bin\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811883 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-etc-tuned\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.812448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.811909 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c739ad-96b5-47dd-8dfa-374af65e3165-tmp\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.818270 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.817963 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:01.818270 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.817985 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:01.818270 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.817995 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:01.818270 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.818053 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:02.318038973 +0000 UTC m=+2.953191081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:01.818270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.818241 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsm7f\" (UniqueName: \"kubernetes.io/projected/fa5d8f7e-3092-496c-a773-1d0ef3c3295f-kube-api-access-bsm7f\") pod \"node-ca-hj8x9\" (UID: \"fa5d8f7e-3092-496c-a773-1d0ef3c3295f\") " pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:01.818809 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.818785 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcjw\" (UniqueName: \"kubernetes.io/projected/e4c739ad-96b5-47dd-8dfa-374af65e3165-kube-api-access-jtcjw\") pod \"tuned-2kwrn\" (UID: \"e4c739ad-96b5-47dd-8dfa-374af65e3165\") " pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:01.820435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.820408 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d68\" (UniqueName: \"kubernetes.io/projected/de262f3f-e229-4776-8132-37278c6f7695-kube-api-access-r4d68\") pod \"iptables-alerter-z5gfr\" (UID: \"de262f3f-e229-4776-8132-37278c6f7695\") " pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:01.820520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.820414 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvfl\" (UniqueName: \"kubernetes.io/projected/3175aee1-79d4-448c-93ea-03d67123e57d-kube-api-access-7hvfl\") pod \"multus-5p8jk\" (UID: \"3175aee1-79d4-448c-93ea-03d67123e57d\") " pod="openshift-multus/multus-5p8jk" Sep 15 21:16:01.820606 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.820589 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlmg\" (UniqueName: \"kubernetes.io/projected/82e87a8f-2d04-4c0e-beb3-2cd0c6b79651-kube-api-access-7zlmg\") pod \"ovnkube-node-qrpkm\" (UID: \"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:01.911980 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.911897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-device-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.911980 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.911941 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dt69\" (UniqueName: \"kubernetes.io/projected/7794745f-2df5-4a52-8ce7-e2d4627909c2-kube-api-access-7dt69\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912012 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-device-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-socket-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912101 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-sys-fs\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912127 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v5c\" (UniqueName: \"kubernetes.io/projected/3187d9d1-bd94-45fd-a140-28094411a7c0-kube-api-access-n6v5c\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912166 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-sys-fs\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912173 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912218 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912241 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912257 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-socket-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.912267 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912319 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:01.912334 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:02.412314912 +0000 UTC m=+3.047467022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912267 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912368 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sll2w\" (UniqueName: \"kubernetes.io/projected/757e5f7e-9388-48af-bc43-6ed61d49a316-kube-api-access-sll2w\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912380 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912391 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912413 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912432 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757e5f7e-9388-48af-bc43-6ed61d49a316-tmp-dir\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912461 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912492 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-system-cni-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912520 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-registration-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912537 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912555 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757e5f7e-9388-48af-bc43-6ed61d49a316-hosts-file\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-cnibin\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912602 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hw8z\" (UniqueName: \"kubernetes.io/projected/e70176b6-167d-40d3-81ff-d27f61b9ad08-kube-api-access-7hw8z\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912643 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757e5f7e-9388-48af-bc43-6ed61d49a316-hosts-file\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912685 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-cnibin\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912730 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7794745f-2df5-4a52-8ce7-e2d4627909c2-system-cni-dir\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912743 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757e5f7e-9388-48af-bc43-6ed61d49a316-tmp-dir\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912738 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3187d9d1-bd94-45fd-a140-28094411a7c0-registration-dir\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.912996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.912965 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.913349 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.913003 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.913349 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.913327 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7794745f-2df5-4a52-8ce7-e2d4627909c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.934921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.934892 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dt69\" (UniqueName: \"kubernetes.io/projected/7794745f-2df5-4a52-8ce7-e2d4627909c2-kube-api-access-7dt69\") pod \"multus-additional-cni-plugins-p7r2c\" (UID: \"7794745f-2df5-4a52-8ce7-e2d4627909c2\") " pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:01.935177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.935146 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sll2w\" (UniqueName: \"kubernetes.io/projected/757e5f7e-9388-48af-bc43-6ed61d49a316-kube-api-access-sll2w\") pod \"node-resolver-fx9kz\" (UID: \"757e5f7e-9388-48af-bc43-6ed61d49a316\") " pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:01.935502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.935485 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hw8z\" (UniqueName: \"kubernetes.io/projected/e70176b6-167d-40d3-81ff-d27f61b9ad08-kube-api-access-7hw8z\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:01.936531 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.936512 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v5c\" (UniqueName: \"kubernetes.io/projected/3187d9d1-bd94-45fd-a140-28094411a7c0-kube-api-access-n6v5c\") pod \"aws-ebs-csi-driver-node-fpfkq\" (UID: \"3187d9d1-bd94-45fd-a140-28094411a7c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:01.997557 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:01.997525 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:02.004247 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.004228 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5p8jk" Sep 15 21:16:02.012420 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.012400 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hj8x9" Sep 15 21:16:02.017940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.017922 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" Sep 15 21:16:02.025658 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.025633 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:02.033505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.033488 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5gfr" Sep 15 21:16:02.038274 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.038258 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fx9kz" Sep 15 21:16:02.044923 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.044902 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" Sep 15 21:16:02.049495 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.049473 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" Sep 15 21:16:02.316004 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.315919 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:02.316171 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.316092 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:02.316171 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.316161 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:03.316144861 +0000 UTC m=+3.951296969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:02.416881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.416819 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.416917 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.416971 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.416997 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.417009 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.417029 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:02.417064 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.417057 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:03.417041971 +0000 UTC m=+4.052194059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:02.417300 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:02.417084 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:03.417065297 +0000 UTC m=+4.052217392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:02.520940 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.520896 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5d8f7e_3092_496c_a773_1d0ef3c3295f.slice/crio-d5ab73dcf6b2cca56034f1eb73012bdffa198afeaf03dd8a95db1596fe5cd3b0 WatchSource:0}: Error finding container d5ab73dcf6b2cca56034f1eb73012bdffa198afeaf03dd8a95db1596fe5cd3b0: Status 404 returned error can't find the container with id d5ab73dcf6b2cca56034f1eb73012bdffa198afeaf03dd8a95db1596fe5cd3b0 Sep 15 21:16:02.523079 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.522975 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757e5f7e_9388_48af_bc43_6ed61d49a316.slice/crio-9729cde872b7842936df9f8eac7b989f42ccc5cd310546f288a3b17deb29c4d3 WatchSource:0}: Error finding container 9729cde872b7842936df9f8eac7b989f42ccc5cd310546f288a3b17deb29c4d3: Status 404 returned error can't find the container with id 9729cde872b7842936df9f8eac7b989f42ccc5cd310546f288a3b17deb29c4d3 Sep 15 21:16:02.528406 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.528385 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3178ee_7869_410c_9852_308038c9bf62.slice/crio-7c32f0b114031b16800d455abdd139dff72c03519dd0ce6d71658cc8e368a2ff WatchSource:0}: Error finding container 7c32f0b114031b16800d455abdd139dff72c03519dd0ce6d71658cc8e368a2ff: Status 404 returned error can't find the container with id 7c32f0b114031b16800d455abdd139dff72c03519dd0ce6d71658cc8e368a2ff Sep 15 21:16:02.529110 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.529049 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c739ad_96b5_47dd_8dfa_374af65e3165.slice/crio-f6b005035f297c7007db6ccdfa5437faf74e52ccf73daba8b3f0b3dbb68564cd WatchSource:0}: Error finding container f6b005035f297c7007db6ccdfa5437faf74e52ccf73daba8b3f0b3dbb68564cd: Status 404 returned error can't find the container with id f6b005035f297c7007db6ccdfa5437faf74e52ccf73daba8b3f0b3dbb68564cd Sep 15 21:16:02.530794 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.530749 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3187d9d1_bd94_45fd_a140_28094411a7c0.slice/crio-f811e141c692f474165589d591a60168b34f4ebbd43feb371fda66f1ba8eadb7 WatchSource:0}: Error finding container f811e141c692f474165589d591a60168b34f4ebbd43feb371fda66f1ba8eadb7: Status 404 returned error can't find the container with id f811e141c692f474165589d591a60168b34f4ebbd43feb371fda66f1ba8eadb7 Sep 15 21:16:02.532756 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.532734 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175aee1_79d4_448c_93ea_03d67123e57d.slice/crio-53c125715eaa5bf5a1dfee12b8f4ab5f5fd149cbac80dc172994c5989dfd88cf WatchSource:0}: Error finding container 53c125715eaa5bf5a1dfee12b8f4ab5f5fd149cbac80dc172994c5989dfd88cf: Status 404 returned error can't find the container with id 53c125715eaa5bf5a1dfee12b8f4ab5f5fd149cbac80dc172994c5989dfd88cf Sep 15 21:16:02.533071 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.533046 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7794745f_2df5_4a52_8ce7_e2d4627909c2.slice/crio-929ab8cc8b43caea97bcc39dec7645b5e787927c84c65cc20a9309c8d0956203 WatchSource:0}: Error finding container 929ab8cc8b43caea97bcc39dec7645b5e787927c84c65cc20a9309c8d0956203: Status 404 returned error can't find the container with id 929ab8cc8b43caea97bcc39dec7645b5e787927c84c65cc20a9309c8d0956203 Sep 15 21:16:02.533444 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.533382 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e87a8f_2d04_4c0e_beb3_2cd0c6b79651.slice/crio-5a411770124e55e39d1f9241e8e697e3eb184c150bec9d549ccd7c4d8e545608 WatchSource:0}: Error finding container 5a411770124e55e39d1f9241e8e697e3eb184c150bec9d549ccd7c4d8e545608: Status 404 returned error can't find the container with id 5a411770124e55e39d1f9241e8e697e3eb184c150bec9d549ccd7c4d8e545608 Sep 15 21:16:02.534764 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.534738 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde262f3f_e229_4776_8132_37278c6f7695.slice/crio-b60a4ca49a3acc4b25c7776e88505113789dafeee11e910dace02a57ba5f2468 WatchSource:0}: Error finding container b60a4ca49a3acc4b25c7776e88505113789dafeee11e910dace02a57ba5f2468: Status 404 returned error can't find the container with id b60a4ca49a3acc4b25c7776e88505113789dafeee11e910dace02a57ba5f2468 Sep 15 21:16:02.540886 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:02.540708 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8484a1774e051d9e3b56ffda320809a.slice/crio-77d7ae689c7525ef75064125da39aefc1f6592e8325a51c8635c69fc5de5423a WatchSource:0}: Error finding container 77d7ae689c7525ef75064125da39aefc1f6592e8325a51c8635c69fc5de5423a: Status 404 returned error can't find the container with id 77d7ae689c7525ef75064125da39aefc1f6592e8325a51c8635c69fc5de5423a Sep 15 21:16:02.745106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.744953 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:11:01 +0000 UTC" deadline="2028-06-24 07:11:49.115924654 +0000 UTC" Sep 15 21:16:02.745106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.745099 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15537h55m46.370829133s" Sep 15 21:16:02.792245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.792211 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"f9572ef115be1ab8679034b1a50b59a2","Type":"ContainerStarted","Data":"b7f1098b01ff087d27f95e4399303f84488e0752cd7b038ae07c20328f9ff02d"} Sep 15 21:16:02.793347 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.793318 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"e8484a1774e051d9e3b56ffda320809a","Type":"ContainerStarted","Data":"77d7ae689c7525ef75064125da39aefc1f6592e8325a51c8635c69fc5de5423a"} Sep 15 21:16:02.794393 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.794367 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5gfr" event={"ID":"de262f3f-e229-4776-8132-37278c6f7695","Type":"ContainerStarted","Data":"b60a4ca49a3acc4b25c7776e88505113789dafeee11e910dace02a57ba5f2468"} Sep 15 21:16:02.795361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.795344 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"5a411770124e55e39d1f9241e8e697e3eb184c150bec9d549ccd7c4d8e545608"} Sep 15 21:16:02.796462 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.796443 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerStarted","Data":"929ab8cc8b43caea97bcc39dec7645b5e787927c84c65cc20a9309c8d0956203"} Sep 15 21:16:02.797491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.797473 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" event={"ID":"e4c739ad-96b5-47dd-8dfa-374af65e3165","Type":"ContainerStarted","Data":"f6b005035f297c7007db6ccdfa5437faf74e52ccf73daba8b3f0b3dbb68564cd"} Sep 15 21:16:02.798523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.798504 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fx9kz" event={"ID":"757e5f7e-9388-48af-bc43-6ed61d49a316","Type":"ContainerStarted","Data":"9729cde872b7842936df9f8eac7b989f42ccc5cd310546f288a3b17deb29c4d3"} Sep 15 21:16:02.799588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.799570 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5p8jk" event={"ID":"3175aee1-79d4-448c-93ea-03d67123e57d","Type":"ContainerStarted","Data":"53c125715eaa5bf5a1dfee12b8f4ab5f5fd149cbac80dc172994c5989dfd88cf"} Sep 15 21:16:02.800581 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.800562 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" event={"ID":"3187d9d1-bd94-45fd-a140-28094411a7c0","Type":"ContainerStarted","Data":"f811e141c692f474165589d591a60168b34f4ebbd43feb371fda66f1ba8eadb7"} Sep 15 21:16:02.804454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.804431 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf44d" event={"ID":"6f3178ee-7869-410c-9852-308038c9bf62","Type":"ContainerStarted","Data":"7c32f0b114031b16800d455abdd139dff72c03519dd0ce6d71658cc8e368a2ff"} Sep 15 21:16:02.805439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:02.805420 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hj8x9" event={"ID":"fa5d8f7e-3092-496c-a773-1d0ef3c3295f","Type":"ContainerStarted","Data":"d5ab73dcf6b2cca56034f1eb73012bdffa198afeaf03dd8a95db1596fe5cd3b0"} Sep 15 21:16:03.324802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.324759 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:03.325122 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.325003 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:03.325122 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.325068 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:05.325049059 +0000 UTC m=+5.960201172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:03.425454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.425421 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:03.425646 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.425500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:03.425706 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425677 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:03.425772 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425734 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:05.42571873 +0000 UTC m=+6.060870820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:03.425860 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425801 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:03.425860 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425820 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:03.425860 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425846 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:03.426004 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.425886 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:05.425872076 +0000 UTC m=+6.061024171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:03.745704 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.745579 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:11:01 +0000 UTC" deadline="2028-06-16 19:30:09.263123412 +0000 UTC" Sep 15 21:16:03.745704 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.745652 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15358h14m5.517475479s" Sep 15 21:16:03.788097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.788066 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:03.788251 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.788198 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:03.788631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.788603 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:03.788743 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.788703 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:03.788816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:03.788763 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:03.788911 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:03.788884 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:05.341143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.341105 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:05.341765 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.341279 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:05.341765 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.341344 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.341325962 +0000 UTC m=+9.976478056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:05.442187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.442151 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:05.442348 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.442229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:05.442421 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442364 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:05.442474 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442424 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.442405858 +0000 UTC m=+10.077557951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:05.442849 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442813 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:05.442849 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442851 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:05.443002 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442865 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:05.443002 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.442909 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.442893339 +0000 UTC m=+10.078045433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:05.784543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.784469 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:05.784741 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.784599 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:05.784938 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.784918 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:05.785017 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:05.784936 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:05.785017 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.785006 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:05.785111 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:05.785069 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:07.785164 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:07.785304 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:07.785721 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:07.785819 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:07.785902 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:07.786005 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:07.785966 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:09.377885 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.377295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:09.377885 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.377470 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:09.377885 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.377529 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.377511089 +0000 UTC m=+18.012663195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.478054 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.478165 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478289 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478316 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478333 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478357 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478362 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.4783428 +0000 UTC m=+18.113494900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:09.478416 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.478390 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.478381261 +0000 UTC m=+18.113533365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.784521 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.784545 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.784648 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.784678 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:09.784713 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:09.785076 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:09.784772 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:11.788312 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:11.788276 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:11.788312 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:11.788276 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:11.788921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:11.788394 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:11.788921 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:11.788424 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:11.788921 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:11.788583 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:11.788921 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:11.788697 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:13.784520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:13.784436 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:13.784520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:13.784482 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:13.785018 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:13.784564 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:13.785018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:13.784595 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:13.785018 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:13.784676 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:13.785018 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:13.784759 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:15.790045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:15.789988 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:15.790496 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:15.790001 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:15.790496 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:15.790130 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:15.790496 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:15.790140 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:15.790496 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:15.790195 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:15.790496 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:15.790286 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:17.436231 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.436152 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:17.436703 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.436296 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:17.436703 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.436356 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:33.436338229 +0000 UTC m=+34.071490318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:17.536732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.536699 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:17.536919 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.536757 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:17.536919 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536873 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:17.536919 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536883 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:17.536919 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536909 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:17.536919 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536923 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:17.537110 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536939 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:33.536920871 +0000 UTC m=+34.172072977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:17.537110 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.536977 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:16:33.536959264 +0000 UTC m=+34.172111373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:17.786982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.786902 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:17.787140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.787014 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:17.787140 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.787016 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:17.787140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:17.787011 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:17.787280 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.787124 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:17.787280 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:17.787236 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:19.789002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:19.788964 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:19.789521 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:19.789049 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:19.789632 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:19.789512 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:19.789632 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:19.789578 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:19.789632 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:19.789084 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:19.789757 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:19.789673 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:21.784262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.783955 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:21.784952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.783955 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:21.784952 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:21.784339 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:21.784952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.784007 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:21.784952 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:21.784415 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:21.784952 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:21.784505 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:21.860980 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.860939 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"ffaefb7c25dab31406981135b8718d460587acda9d66d778e99ad331b60d03ed"} Sep 15 21:16:21.861210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.861185 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"49ffa3a61404e37a1423511e09878cae6a954307a99eb84697f149acb2c3c2ba"} Sep 15 21:16:21.861327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.861246 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"54b5ed5461f68d4cae3ebd3ed1b3e89105146edba42efd1e160de523282626b9"} Sep 15 21:16:21.861327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.861282 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"68684e296e08c7def45bc9802c36284ab4040bc9b4d05ae377095ff3b143b8fc"} Sep 15 21:16:21.861437 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.861327 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"66176d5ae0af37e074dd565cf8bd225a4c0049b684496dd8f288c19ac3f5f542"} Sep 15 21:16:21.861437 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.861375 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"4250d486ea1be9c1ccab475a1a47dd0f83741769bea4491ea0b409aed64bc910"} Sep 15 21:16:21.862770 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.862740 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="bee2c4f3083a87398a7498aed99ef5d75fadd6ed7596909a05247e00f042f8da" exitCode=0 Sep 15 21:16:21.862925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.862822 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"bee2c4f3083a87398a7498aed99ef5d75fadd6ed7596909a05247e00f042f8da"} Sep 15 21:16:21.864641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.864618 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" event={"ID":"e4c739ad-96b5-47dd-8dfa-374af65e3165","Type":"ContainerStarted","Data":"c4451c139c740386260ddfec6f227cf52537b0a55d0da7e8dae2042a28d23430"} Sep 15 21:16:21.867846 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.867798 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fx9kz" event={"ID":"757e5f7e-9388-48af-bc43-6ed61d49a316","Type":"ContainerStarted","Data":"e4166d8c472c0e996a6d37437e9911bc5df96679be654cbe7d758378ccf8bca3"} Sep 15 21:16:21.870142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.870114 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5p8jk" event={"ID":"3175aee1-79d4-448c-93ea-03d67123e57d","Type":"ContainerStarted","Data":"907bdb6ab76738e4d0638ea4a6f54e8129eb9dbec5c0b50bfcf2f6f6586605fc"} Sep 15 21:16:21.871947 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.871897 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" event={"ID":"3187d9d1-bd94-45fd-a140-28094411a7c0","Type":"ContainerStarted","Data":"9d17ed4f5cc4b21480ec9591a9a81835b11c29e84ec36942285d8f86411605ac"} Sep 15 21:16:21.873390 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.873344 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf44d" event={"ID":"6f3178ee-7869-410c-9852-308038c9bf62","Type":"ContainerStarted","Data":"df586b9ce00c3665c088b693bca676f7366d7ed7288eb7b0dac7f6c2dd0378c8"} Sep 15 21:16:21.874913 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.874896 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hj8x9" event={"ID":"fa5d8f7e-3092-496c-a773-1d0ef3c3295f","Type":"ContainerStarted","Data":"c2a0febdd714003ef90941d9428d3e4045af652643946df0b7eb6eba772c64e3"} Sep 15 21:16:21.876429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.876401 2552 generic.go:356] "Generic (PLEG): container finished" podID="e8484a1774e051d9e3b56ffda320809a" containerID="e51762351bd5e4821f345281f98dc3db05245121182b08577f7daa8e8f94ab89" exitCode=0 Sep 15 21:16:21.876429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.876427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"e8484a1774e051d9e3b56ffda320809a","Type":"ContainerDied","Data":"e51762351bd5e4821f345281f98dc3db05245121182b08577f7daa8e8f94ab89"} Sep 15 21:16:21.887647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.887593 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" podStartSLOduration=20.887576665 podStartE2EDuration="20.887576665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:02.80660231 +0000 UTC m=+3.441754430" watchObservedRunningTime="2026-09-15 21:16:21.887576665 +0000 UTC m=+22.522728779" Sep 15 21:16:21.909856 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.909802 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hj8x9" podStartSLOduration=4.723840396 podStartE2EDuration="22.909786302s" totalImagesPullingTime="18.185945906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.909436502 +0000 UTC m=+22.544588613" watchObservedRunningTime="2026-09-15 21:16:21.909786302 +0000 UTC m=+22.544938414" Sep 15 21:16:21.953432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.953380 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vf44d" podStartSLOduration=3.8068715859999998 podStartE2EDuration="21.953359042s" totalImagesPullingTime="18.146487456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.952993885 +0000 UTC m=+22.588146001" watchObservedRunningTime="2026-09-15 21:16:21.953359042 +0000 UTC m=+22.588511155" Sep 15 21:16:21.953575 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.953514 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2kwrn" podStartSLOduration=4.774960909 podStartE2EDuration="22.953505116s" totalImagesPullingTime="18.178544207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.934015836 +0000 UTC m=+22.569167963" watchObservedRunningTime="2026-09-15 21:16:21.953505116 +0000 UTC m=+22.588657229" Sep 15 21:16:21.970046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.969999 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fx9kz" podStartSLOduration=3.805925125 podStartE2EDuration="21.9699817s" totalImagesPullingTime="18.164056575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.969624783 +0000 UTC m=+22.604776894" watchObservedRunningTime="2026-09-15 21:16:21.9699817 +0000 UTC m=+22.605133812" Sep 15 21:16:21.993254 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:21.993208 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5p8jk" podStartSLOduration=4.596730099 podStartE2EDuration="22.993189599s" totalImagesPullingTime="18.3964595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:21.992676361 +0000 UTC m=+22.627828484" watchObservedRunningTime="2026-09-15 21:16:21.993189599 +0000 UTC m=+22.628341711" Sep 15 21:16:22.654042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.654019 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:22.667054 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.667033 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 21:16:22.771813 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.771679 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:16:22.667048215Z","UUID":"bff8b87a-3d8d-4712-a01b-5aeee84e5d60","Handler":null,"Name":"","Endpoint":""} Sep 15 21:16:22.773099 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.773080 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 21:16:22.773199 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.773112 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 21:16:22.880349 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.880276 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"e8484a1774e051d9e3b56ffda320809a","Type":"ContainerStarted","Data":"80434834d06c2ab884eb4c6b10ce9a2a5e7c92c08fe42216c44393bee16fc3f4"} Sep 15 21:16:22.881720 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.881694 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5gfr" event={"ID":"de262f3f-e229-4776-8132-37278c6f7695","Type":"ContainerStarted","Data":"a9a4889891e7526a9621283499256b0c653f6deaea9f57d5d7a748d462422fce"} Sep 15 21:16:22.883450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.883427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" event={"ID":"3187d9d1-bd94-45fd-a140-28094411a7c0","Type":"ContainerStarted","Data":"872b9a37039e3a0399c565a50daee1721f9006078b9005518f99c0f5dc96f341"} Sep 15 21:16:22.901706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.901662 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" podStartSLOduration=21.901650167 podStartE2EDuration="21.901650167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:22.901621014 +0000 UTC m=+23.536773126" watchObservedRunningTime="2026-09-15 21:16:22.901650167 +0000 UTC m=+23.536802256" Sep 15 21:16:22.935998 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:22.935956 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z5gfr" podStartSLOduration=4.758688406 podStartE2EDuration="22.935944662s" totalImagesPullingTime="18.177256256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:22.935893725 +0000 UTC m=+23.571045836" watchObservedRunningTime="2026-09-15 21:16:22.935944662 +0000 UTC m=+23.571096773" Sep 15 21:16:23.784460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:23.784284 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:23.784554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:23.784284 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:23.784617 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:23.784549 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:23.784617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:23.784288 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:23.784617 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:23.784605 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:23.784766 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:23.784682 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:23.893168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:23.893135 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"0e864dfd491a0b5afa3745b031e52ace6823abe2cb546ea3e7e0459833cd632c"} Sep 15 21:16:24.640890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:24.640852 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:24.641475 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:24.641459 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:24.897904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:24.897799 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" event={"ID":"3187d9d1-bd94-45fd-a140-28094411a7c0","Type":"ContainerStarted","Data":"2679d7c026df4653a6bd2d10dde642d5f88b66b68b98b7eefe5030da243816f8"} Sep 15 21:16:24.898446 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:24.898429 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vf44d" Sep 15 21:16:24.972415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:24.972367 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpfkq" podStartSLOduration=4.442070252 podStartE2EDuration="24.97235047s" totalImagesPullingTime="20.530280218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:24.936443666 +0000 UTC m=+25.571595777" watchObservedRunningTime="2026-09-15 21:16:24.97235047 +0000 UTC m=+25.607502582" Sep 15 21:16:25.784334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:25.784303 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:25.784334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:25.784337 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:25.784520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:25.784303 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:25.784520 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:25.784408 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:25.784520 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:25.784494 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:25.784617 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:25.784571 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:26.906645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:26.906444 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" event={"ID":"82e87a8f-2d04-4c0e-beb3-2cd0c6b79651","Type":"ContainerStarted","Data":"354ef78e0a134b8e5874eb74da2eb763979f57fdaaefacfb0816df309304cde9"} Sep 15 21:16:26.935727 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:26.935666 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" podStartSLOduration=9.542845349 podStartE2EDuration="27.935645444s" totalImagesPullingTime="18.392800095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:26.935356127 +0000 UTC m=+27.570508238" watchObservedRunningTime="2026-09-15 21:16:26.935645444 +0000 UTC m=+27.570797560" Sep 15 21:16:27.784341 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.784310 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:27.784341 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.784332 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:27.784560 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.784310 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:27.784560 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.784448 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:27.784662 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.784590 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:27.784711 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.784677 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:27.885450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.885408 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fzbx"] Sep 15 21:16:27.886314 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.886284 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2l6s"] Sep 15 21:16:27.886857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.886819 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6w6wz"] Sep 15 21:16:27.908319 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908257 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908262 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908262 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.908378 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908388 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.908486 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908528 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.908580 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:27.909049 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:27.908667 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:27.927508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.927480 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:27.927655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:27.927630 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:16:29.785231 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:29.785206 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:29.785760 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:29.785320 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:29.785760 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:29.785387 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:29.785760 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:29.785479 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:29.785760 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:29.785515 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:29.785760 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:29.785575 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:29.913410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:29.913378 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="1c76f6e075348359108bb6d173f860f1a0c0bedaa11ce0878a5f4b44631225f4" exitCode=0 Sep 15 21:16:29.913591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:29.913464 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"1c76f6e075348359108bb6d173f860f1a0c0bedaa11ce0878a5f4b44631225f4"} Sep 15 21:16:31.784568 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:31.784392 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:31.784956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:31.784401 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:31.784956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:31.784401 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:31.784956 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:31.784758 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fzbx" podUID="e70176b6-167d-40d3-81ff-d27f61b9ad08" Sep 15 21:16:31.784956 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:31.784815 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2l6s" podUID="cd7ffb29-5e39-4f8b-a217-ad34ede637ce" Sep 15 21:16:31.784956 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:31.784643 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w6wz" podUID="0487501f-1be4-4d51-8ba4-8314af778987" Sep 15 21:16:31.919744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:31.919714 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="f07319a566d29c50499f09f1e6c98bf4b3b03ceb3234a51f5cdda7e414678fe9" exitCode=0 Sep 15 21:16:31.919919 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:31.919768 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"f07319a566d29c50499f09f1e6c98bf4b3b03ceb3234a51f5cdda7e414678fe9"} Sep 15 21:16:33.455538 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.455509 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:33.455886 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.455618 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:33.455886 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.455667 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret podName:0487501f-1be4-4d51-8ba4-8314af778987 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:05.455653599 +0000 UTC m=+66.090805688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret") pod "global-pull-secret-syncer-6w6wz" (UID: "0487501f-1be4-4d51-8ba4-8314af778987") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:33.556123 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.556095 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:33.556259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.556143 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:33.556259 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556233 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:33.556323 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556253 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:33.556323 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556277 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:33.556323 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556282 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs podName:e70176b6-167d-40d3-81ff-d27f61b9ad08 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:05.556267434 +0000 UTC m=+66.191419524 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs") pod "network-metrics-daemon-9fzbx" (UID: "e70176b6-167d-40d3-81ff-d27f61b9ad08") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:33.556323 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556286 2552 projected.go:196] Error preparing data for projected volume kube-api-access-2qmgx for pod openshift-network-diagnostics/network-check-target-m2l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:33.556463 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.556328 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx podName:cd7ffb29-5e39-4f8b-a217-ad34ede637ce nodeName:}" failed. No retries permitted until 2026-09-15 21:17:05.556315882 +0000 UTC m=+66.191467971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qmgx" (UniqueName: "kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx") pod "network-check-target-m2l6s" (UID: "cd7ffb29-5e39-4f8b-a217-ad34ede637ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:33.655696 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.655671 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeReady" Sep 15 21:16:33.655919 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.655793 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 21:16:33.710654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.710585 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vwjjx"] Sep 15 21:16:33.723062 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.722971 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fjdlx"] Sep 15 21:16:33.723409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.723393 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.727093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.727069 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ztstg\"" Sep 15 21:16:33.727306 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.727097 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 21:16:33.728670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.728652 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 21:16:33.735102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.735082 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vwjjx"] Sep 15 21:16:33.735182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.735116 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fjdlx"] Sep 15 21:16:33.735220 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.735207 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.739143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.739122 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 21:16:33.739393 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.739378 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 21:16:33.739435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.739410 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 21:16:33.745309 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.745290 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k7wrd\"" Sep 15 21:16:33.784464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.784441 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:16:33.784637 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.784452 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:16:33.784637 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.784555 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:16:33.786918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.786898 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\"" Sep 15 21:16:33.787030 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.786937 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 21:16:33.787030 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.787006 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\"" Sep 15 21:16:33.787169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.787133 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:16:33.787302 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.787177 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:16:33.787302 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.787219 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 21:16:33.858077 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858050 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9238defc-83a1-40db-9a48-7002cd3a4010-config-volume\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.858249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858102 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqpz\" (UniqueName: \"kubernetes.io/projected/c71e5357-0147-4721-a31c-09e2e67ce367-kube-api-access-szqpz\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.858249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858119 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwvf\" (UniqueName: \"kubernetes.io/projected/9238defc-83a1-40db-9a48-7002cd3a4010-kube-api-access-wpwvf\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.858249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858135 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9238defc-83a1-40db-9a48-7002cd3a4010-tmp-dir\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.858249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71e5357-0147-4721-a31c-09e2e67ce367-cert\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.858249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.858202 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.927871 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.927839 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="38589cc9555ce2305a0bf421cdf259212f4c2d3b10091223513d6f121c3714b9" exitCode=0 Sep 15 21:16:33.928012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.927893 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"38589cc9555ce2305a0bf421cdf259212f4c2d3b10091223513d6f121c3714b9"} Sep 15 21:16:33.959232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959198 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqpz\" (UniqueName: \"kubernetes.io/projected/c71e5357-0147-4721-a31c-09e2e67ce367-kube-api-access-szqpz\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.959232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwvf\" (UniqueName: \"kubernetes.io/projected/9238defc-83a1-40db-9a48-7002cd3a4010-kube-api-access-wpwvf\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.959408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959244 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9238defc-83a1-40db-9a48-7002cd3a4010-tmp-dir\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.959408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959262 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71e5357-0147-4721-a31c-09e2e67ce367-cert\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.959408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959283 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.959408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959336 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9238defc-83a1-40db-9a48-7002cd3a4010-config-volume\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.959408 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.959367 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:33.959688 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:33.959414 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:34.459397682 +0000 UTC m=+35.094549771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:33.959688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959620 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9238defc-83a1-40db-9a48-7002cd3a4010-tmp-dir\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.959883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.959863 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9238defc-83a1-40db-9a48-7002cd3a4010-config-volume\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.963680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.963632 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71e5357-0147-4721-a31c-09e2e67ce367-cert\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:33.969695 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.969675 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwvf\" (UniqueName: \"kubernetes.io/projected/9238defc-83a1-40db-9a48-7002cd3a4010-kube-api-access-wpwvf\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:33.969853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:33.969808 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqpz\" (UniqueName: \"kubernetes.io/projected/c71e5357-0147-4721-a31c-09e2e67ce367-kube-api-access-szqpz\") pod \"ingress-canary-fjdlx\" (UID: \"c71e5357-0147-4721-a31c-09e2e67ce367\") " pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:34.043878 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:34.043848 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fjdlx" Sep 15 21:16:34.343435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:34.343357 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fjdlx"] Sep 15 21:16:34.348292 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:34.348259 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71e5357_0147_4721_a31c_09e2e67ce367.slice/crio-eb3f6d0d0d48444a60475dc9477bb6c5059f16bdc97cd076e8c8f782bedb7d22 WatchSource:0}: Error finding container eb3f6d0d0d48444a60475dc9477bb6c5059f16bdc97cd076e8c8f782bedb7d22: Status 404 returned error can't find the container with id eb3f6d0d0d48444a60475dc9477bb6c5059f16bdc97cd076e8c8f782bedb7d22 Sep 15 21:16:34.462872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:34.462822 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:34.463353 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:34.462957 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:34.463353 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:34.463027 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:35.463006005 +0000 UTC m=+36.098158107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:34.932102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:34.932057 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fjdlx" event={"ID":"c71e5357-0147-4721-a31c-09e2e67ce367","Type":"ContainerStarted","Data":"eb3f6d0d0d48444a60475dc9477bb6c5059f16bdc97cd076e8c8f782bedb7d22"} Sep 15 21:16:35.472032 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:35.471983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:35.472437 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:35.472164 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:35.472437 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:35.472218 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.472201347 +0000 UTC m=+38.107353436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:36.956169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.956132 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn"] Sep 15 21:16:36.963144 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.963112 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" Sep 15 21:16:36.965521 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.965497 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:36.965836 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.965798 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:36.965911 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.965820 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pnw6m\"" Sep 15 21:16:36.969437 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:36.969404 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn"] Sep 15 21:16:37.061583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.061548 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js"] Sep 15 21:16:37.071207 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.071178 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.072348 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.072317 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2ss7v"] Sep 15 21:16:37.075128 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.075108 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.075236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.075171 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 21:16:37.075853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.075684 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mgbts\"" Sep 15 21:16:37.075853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.075789 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 21:16:37.075853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.075850 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.081711 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.081693 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.082615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.082274 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cdd487d68-4972n"] Sep 15 21:16:37.085095 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.085070 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlcq\" (UniqueName: \"kubernetes.io/projected/e6618063-1fe3-46a2-a8b2-16e049b31415-kube-api-access-jvlcq\") pod \"volume-data-source-validator-86695b7cc7-nnvbn\" (UID: \"e6618063-1fe3-46a2-a8b2-16e049b31415\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" Sep 15 21:16:37.087065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.086818 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 21:16:37.087065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.086907 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.087065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.086957 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2mxxg\"" Sep 15 21:16:37.087065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.086994 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 21:16:37.087342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.087224 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.093991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.093971 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-b24hp"] Sep 15 21:16:37.094083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.094072 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.103928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.103718 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 15 21:16:37.103928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.103759 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 15 21:16:37.103928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.103812 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 15 21:16:37.105332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.104873 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.105332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.105110 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.105474 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.105379 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qtspj\"" Sep 15 21:16:37.105474 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.105393 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 15 21:16:37.109340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.108993 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js"] Sep 15 21:16:37.109340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.109035 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2ss7v"] Sep 15 21:16:37.110452 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.110427 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.112454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.112434 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-b24hp"] Sep 15 21:16:37.112589 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.112440 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 21:16:37.114017 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.113996 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cdd487d68-4972n"] Sep 15 21:16:37.114326 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.114308 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 15 21:16:37.114972 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.114950 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.115297 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.115228 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 15 21:16:37.115694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.115449 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whddh\"" Sep 15 21:16:37.115984 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.115965 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.121915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.121898 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 15 21:16:37.166417 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.166389 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv"] Sep 15 21:16:37.170905 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.170879 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" Sep 15 21:16:37.173523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.173498 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c85v7\"" Sep 15 21:16:37.185545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185523 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.185680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185560 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-config\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.185680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185629 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlcq\" (UniqueName: \"kubernetes.io/projected/e6618063-1fe3-46a2-a8b2-16e049b31415-kube-api-access-jvlcq\") pod \"volume-data-source-validator-86695b7cc7-nnvbn\" (UID: \"e6618063-1fe3-46a2-a8b2-16e049b31415\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" Sep 15 21:16:37.185680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185661 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxzj\" (UniqueName: \"kubernetes.io/projected/637e59d0-4000-4180-a0ca-92c166ac1061-kube-api-access-cnxzj\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.185802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185684 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmnd\" (UniqueName: \"kubernetes.io/projected/c716d930-781b-428b-b942-0129be7d46fe-kube-api-access-ltmnd\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.185802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185715 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbtz\" (UniqueName: \"kubernetes.io/projected/2dedd29e-1f9c-4549-8697-44de0145c8e6-kube-api-access-7jbtz\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.185802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185736 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-tmp\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.185802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185762 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-stats-auth\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.185802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185786 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185810 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c716d930-781b-428b-b942-0129be7d46fe-serving-cert\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185858 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-service-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185892 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/637e59d0-4000-4180-a0ca-92c166ac1061-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185919 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-trusted-ca\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185971 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-default-certificate\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.186016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.185995 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-trusted-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.186282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.186017 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-serving-cert\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.186282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.186268 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.186378 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.186295 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-snapshots\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.186432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.186397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87wpb\" (UniqueName: \"kubernetes.io/projected/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-kube-api-access-87wpb\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.187130 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.187073 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv"] Sep 15 21:16:37.208983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.208931 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlcq\" (UniqueName: \"kubernetes.io/projected/e6618063-1fe3-46a2-a8b2-16e049b31415-kube-api-access-jvlcq\") pod \"volume-data-source-validator-86695b7cc7-nnvbn\" (UID: \"e6618063-1fe3-46a2-a8b2-16e049b31415\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" Sep 15 21:16:37.275239 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.275212 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" Sep 15 21:16:37.287559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287532 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.287704 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287564 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-config\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.287704 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287597 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqs6\" (UniqueName: \"kubernetes.io/projected/d8afb728-416e-4d22-a63f-bb14feb28147-kube-api-access-ngqs6\") pod \"network-check-source-6f8b768455-7vlzv\" (UID: \"d8afb728-416e-4d22-a63f-bb14feb28147\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" Sep 15 21:16:37.287704 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287661 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxzj\" (UniqueName: \"kubernetes.io/projected/637e59d0-4000-4180-a0ca-92c166ac1061-kube-api-access-cnxzj\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.287704 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.287682 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 21:16:37.287900 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287686 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmnd\" (UniqueName: \"kubernetes.io/projected/c716d930-781b-428b-b942-0129be7d46fe-kube-api-access-ltmnd\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.287900 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.287765 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.787742732 +0000 UTC m=+38.422894823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : secret "router-metrics-certs-default" not found Sep 15 21:16:37.287900 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287820 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbtz\" (UniqueName: \"kubernetes.io/projected/2dedd29e-1f9c-4549-8697-44de0145c8e6-kube-api-access-7jbtz\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.287900 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287862 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-tmp\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.287900 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287891 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-stats-auth\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287917 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287943 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c716d930-781b-428b-b942-0129be7d46fe-serving-cert\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.287972 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-service-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288001 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/637e59d0-4000-4180-a0ca-92c166ac1061-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.288024 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.288074 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.788059049 +0000 UTC m=+38.423211152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288027 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-trusted-ca\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.288153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288147 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-default-certificate\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288182 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-trusted-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288214 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-serving-cert\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288263 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-snapshots\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288313 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-tmp\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288375 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87wpb\" (UniqueName: \"kubernetes.io/projected/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-kube-api-access-87wpb\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.288544 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288453 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-config\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.288891 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.288730 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.788711573 +0000 UTC m=+38.423863678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:37.288891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.288866 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-service-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.289356 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.289290 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-snapshots\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.289472 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.289452 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c716d930-781b-428b-b942-0129be7d46fe-trusted-ca\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.289738 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.289634 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/637e59d0-4000-4180-a0ca-92c166ac1061-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.290118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.290077 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-trusted-ca-bundle\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.291204 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.291181 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-serving-cert\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.291387 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.291360 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-default-certificate\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.291493 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.291428 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c716d930-781b-428b-b942-0129be7d46fe-serving-cert\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.292214 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.292192 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-stats-auth\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.308204 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.308179 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmnd\" (UniqueName: \"kubernetes.io/projected/c716d930-781b-428b-b942-0129be7d46fe-kube-api-access-ltmnd\") pod \"console-operator-696884fb9c-2ss7v\" (UID: \"c716d930-781b-428b-b942-0129be7d46fe\") " pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.308361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.308337 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxzj\" (UniqueName: \"kubernetes.io/projected/637e59d0-4000-4180-a0ca-92c166ac1061-kube-api-access-cnxzj\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.311076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.311053 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbtz\" (UniqueName: \"kubernetes.io/projected/2dedd29e-1f9c-4549-8697-44de0145c8e6-kube-api-access-7jbtz\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.315274 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.315255 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87wpb\" (UniqueName: \"kubernetes.io/projected/bd64bcee-baf1-4cf4-93fd-4c0a041dd634-kube-api-access-87wpb\") pod \"insights-operator-5844b856db-b24hp\" (UID: \"bd64bcee-baf1-4cf4-93fd-4c0a041dd634\") " pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.320003 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.319981 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:16:37.330357 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.330336 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.333042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.333019 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 21:16:37.333575 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.333558 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 21:16:37.333677 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.333591 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 21:16:37.333860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.333844 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzsgv\"" Sep 15 21:16:37.339846 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.339802 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 21:16:37.346475 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.346451 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:16:37.389735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.389701 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqs6\" (UniqueName: \"kubernetes.io/projected/d8afb728-416e-4d22-a63f-bb14feb28147-kube-api-access-ngqs6\") pod \"network-check-source-6f8b768455-7vlzv\" (UID: \"d8afb728-416e-4d22-a63f-bb14feb28147\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" Sep 15 21:16:37.393961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.393931 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:37.399100 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.399076 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqs6\" (UniqueName: \"kubernetes.io/projected/d8afb728-416e-4d22-a63f-bb14feb28147-kube-api-access-ngqs6\") pod \"network-check-source-6f8b768455-7vlzv\" (UID: \"d8afb728-416e-4d22-a63f-bb14feb28147\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" Sep 15 21:16:37.423634 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.423613 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-b24hp" Sep 15 21:16:37.481259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.481175 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" Sep 15 21:16:37.490812 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490782 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnv9\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.490937 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490859 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.490937 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490894 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.490937 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:37.491075 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490961 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.491075 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.490980 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.491075 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.491016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.491166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.491094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.491166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.491135 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.491166 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.491105 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:37.491252 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.491200 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:41.491187854 +0000 UTC m=+42.126339944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:37.591591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591549 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.591772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591647 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.591772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591680 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.591772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591725 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591775 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591845 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591898 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnv9\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.591937 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592193 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.592120 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.592285 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.592258 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:37.592344 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.592287 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58cb5bd455-nzfms: secret "image-registry-tls" not found Sep 15 21:16:37.592386 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.592362 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls podName:b58dc70f-4553-4fcb-9e80-346edd6e4d40 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.0923407 +0000 UTC m=+38.727492806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls") pod "image-registry-58cb5bd455-nzfms" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40") : secret "image-registry-tls" not found Sep 15 21:16:37.592930 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.592886 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.593042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.592969 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.594672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.594648 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.594782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.594673 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.603080 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.603053 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.603276 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.603257 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnv9\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:37.794180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.794101 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.794342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.794193 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:37.794342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:37.794270 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:37.794468 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.794348 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.794327464 +0000 UTC m=+39.429479568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:37.794468 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.794346 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:37.794468 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.794396 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.794386202 +0000 UTC m=+39.429538295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:37.794902 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.794884 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 21:16:37.794960 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:37.794949 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.794930899 +0000 UTC m=+39.430083002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : secret "router-metrics-certs-default" not found Sep 15 21:16:38.096777 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:38.096691 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:38.097295 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.096874 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:38.097295 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.096894 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58cb5bd455-nzfms: secret "image-registry-tls" not found Sep 15 21:16:38.097295 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.096961 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls podName:b58dc70f-4553-4fcb-9e80-346edd6e4d40 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:39.096945375 +0000 UTC m=+39.732097464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls") pod "image-registry-58cb5bd455-nzfms" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40") : secret "image-registry-tls" not found Sep 15 21:16:38.802880 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:38.802814 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:38.803069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:38.802923 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:38.803069 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.802966 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:38.803069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:38.802987 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:38.803069 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.803031 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:40.803011176 +0000 UTC m=+41.438163307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:38.803069 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.803045 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 21:16:38.803249 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.803093 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:40.803079538 +0000 UTC m=+41.438231626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : secret "router-metrics-certs-default" not found Sep 15 21:16:38.803249 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:38.803111 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:40.803102238 +0000 UTC m=+41.438254332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:39.107435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.107276 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:39.107896 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:39.107454 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:39.107896 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:39.107475 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58cb5bd455-nzfms: secret "image-registry-tls" not found Sep 15 21:16:39.107896 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:39.107537 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls podName:b58dc70f-4553-4fcb-9e80-346edd6e4d40 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:41.107517744 +0000 UTC m=+41.742669850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls") pod "image-registry-58cb5bd455-nzfms" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40") : secret "image-registry-tls" not found Sep 15 21:16:39.224958 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.224923 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2ss7v"] Sep 15 21:16:39.227609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.227534 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv"] Sep 15 21:16:39.228918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.228854 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn"] Sep 15 21:16:39.229918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.229896 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-b24hp"] Sep 15 21:16:39.243886 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:39.243768 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6618063_1fe3_46a2_a8b2_16e049b31415.slice/crio-b5ccd5737e37b32825ad3c2394468e2f3319c20cdd3534d07138065fcc219d44 WatchSource:0}: Error finding container b5ccd5737e37b32825ad3c2394468e2f3319c20cdd3534d07138065fcc219d44: Status 404 returned error can't find the container with id b5ccd5737e37b32825ad3c2394468e2f3319c20cdd3534d07138065fcc219d44 Sep 15 21:16:39.244636 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:39.244337 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8afb728_416e_4d22_a63f_bb14feb28147.slice/crio-24e90167f1da202648099631d507c8da014ecc0e87003bf1e94a989a1e8f0dae WatchSource:0}: Error finding container 24e90167f1da202648099631d507c8da014ecc0e87003bf1e94a989a1e8f0dae: Status 404 returned error can't find the container with id 24e90167f1da202648099631d507c8da014ecc0e87003bf1e94a989a1e8f0dae Sep 15 21:16:39.244808 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:39.244781 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd64bcee_baf1_4cf4_93fd_4c0a041dd634.slice/crio-241ff2ef3d0850fc5e47eac9ad45690fc75744b827f3c43be4d6ecc827e6a140 WatchSource:0}: Error finding container 241ff2ef3d0850fc5e47eac9ad45690fc75744b827f3c43be4d6ecc827e6a140: Status 404 returned error can't find the container with id 241ff2ef3d0850fc5e47eac9ad45690fc75744b827f3c43be4d6ecc827e6a140 Sep 15 21:16:39.949713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.948979 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="73a2c281470b09fd5641d0ec4a70435036a7b1c4b556e427d3b3f66afd38b475" exitCode=0 Sep 15 21:16:39.949713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.949098 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"73a2c281470b09fd5641d0ec4a70435036a7b1c4b556e427d3b3f66afd38b475"} Sep 15 21:16:39.952035 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.951999 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fjdlx" event={"ID":"c71e5357-0147-4721-a31c-09e2e67ce367","Type":"ContainerStarted","Data":"54e51228dbbb8ce21ba955640bfb76771aec0cd8459c9579598ceba0d1f80452"} Sep 15 21:16:39.954200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.954140 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" event={"ID":"e6618063-1fe3-46a2-a8b2-16e049b31415","Type":"ContainerStarted","Data":"b5ccd5737e37b32825ad3c2394468e2f3319c20cdd3534d07138065fcc219d44"} Sep 15 21:16:39.955915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.955865 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-b24hp" event={"ID":"bd64bcee-baf1-4cf4-93fd-4c0a041dd634","Type":"ContainerStarted","Data":"241ff2ef3d0850fc5e47eac9ad45690fc75744b827f3c43be4d6ecc827e6a140"} Sep 15 21:16:39.958072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.958014 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" event={"ID":"d8afb728-416e-4d22-a63f-bb14feb28147","Type":"ContainerStarted","Data":"24e90167f1da202648099631d507c8da014ecc0e87003bf1e94a989a1e8f0dae"} Sep 15 21:16:39.959283 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.959258 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" event={"ID":"c716d930-781b-428b-b942-0129be7d46fe","Type":"ContainerStarted","Data":"d30fde92a7d8869b999c5136922b9d3a7d59b57318f42ab7345f882342b70f4c"} Sep 15 21:16:39.987875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:39.987805 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fjdlx" podStartSLOduration=2.402330553 podStartE2EDuration="6.987785045s" totalImagesPullingTime="4.585454492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:39.987204271 +0000 UTC m=+40.622356379" watchObservedRunningTime="2026-09-15 21:16:39.987785045 +0000 UTC m=+40.622937157" Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:40.824157 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:40.824255 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:40.824288 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:40.824340 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:40.824414 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.824397345 +0000 UTC m=+45.459549439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : secret "router-metrics-certs-default" not found Sep 15 21:16:40.824451 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:40.824432 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.824422529 +0000 UTC m=+45.459574619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:40.825196 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:40.824459 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:40.825196 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:40.824530 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.824512891 +0000 UTC m=+45.459664995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:40.964455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:40.964425 2552 generic.go:356] "Generic (PLEG): container finished" podID="7794745f-2df5-4a52-8ce7-e2d4627909c2" containerID="14f90d900167efe5cc90f87563417e6fed4b98f16448dfc354af6f45f5065eb4" exitCode=0 Sep 15 21:16:40.964611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:40.964502 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerDied","Data":"14f90d900167efe5cc90f87563417e6fed4b98f16448dfc354af6f45f5065eb4"} Sep 15 21:16:41.126999 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:41.126956 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:41.127148 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:41.127078 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:41.127148 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:41.127100 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58cb5bd455-nzfms: secret "image-registry-tls" not found Sep 15 21:16:41.127260 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:41.127212 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls podName:b58dc70f-4553-4fcb-9e80-346edd6e4d40 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.1271919 +0000 UTC m=+45.762343992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls") pod "image-registry-58cb5bd455-nzfms" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40") : secret "image-registry-tls" not found Sep 15 21:16:41.531111 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:41.531077 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:41.531262 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:41.531238 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:41.531320 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:41.531307 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:49.531287761 +0000 UTC m=+50.166439855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:42.972535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.972498 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/0.log" Sep 15 21:16:42.972961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.972551 2552 generic.go:356] "Generic (PLEG): container finished" podID="c716d930-781b-428b-b942-0129be7d46fe" containerID="bf1646c49c34057ae24f56cb05d1768ee8de2b5d6db20b5e10846f7e51454b6b" exitCode=255 Sep 15 21:16:42.972961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.972643 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" event={"ID":"c716d930-781b-428b-b942-0129be7d46fe","Type":"ContainerDied","Data":"bf1646c49c34057ae24f56cb05d1768ee8de2b5d6db20b5e10846f7e51454b6b"} Sep 15 21:16:42.972961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.972910 2552 scope.go:117] "RemoveContainer" containerID="bf1646c49c34057ae24f56cb05d1768ee8de2b5d6db20b5e10846f7e51454b6b" Sep 15 21:16:42.977168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.977145 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" event={"ID":"7794745f-2df5-4a52-8ce7-e2d4627909c2","Type":"ContainerStarted","Data":"8867edb9958493d4561b6d970b94a7576aaa674d2e260dbe49cb4435cb2e9bb5"} Sep 15 21:16:42.978981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.978959 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" event={"ID":"e6618063-1fe3-46a2-a8b2-16e049b31415","Type":"ContainerStarted","Data":"4ce10b3cf9667cedbdad39d8f01ce5109264b1688299e8f802bfa25c0c08cabf"} Sep 15 21:16:42.980838 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:42.980779 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-b24hp" event={"ID":"bd64bcee-baf1-4cf4-93fd-4c0a041dd634","Type":"ContainerStarted","Data":"cb872878e96d06df7c721ba7efe21b33fa06e80b0cab4d0a469048434cca47d4"} Sep 15 21:16:43.031258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.031212 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-b24hp" podStartSLOduration=2.686477743 podStartE2EDuration="6.031194661s" totalImagesPullingTime="3.344716918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:43.029195898 +0000 UTC m=+43.664348017" watchObservedRunningTime="2026-09-15 21:16:43.031194661 +0000 UTC m=+43.666346766" Sep 15 21:16:43.089791 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.089735 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p7r2c" podStartSLOduration=10.423768326 podStartE2EDuration="43.089712129s" totalImagesPullingTime="32.665943803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:43.087695581 +0000 UTC m=+43.722847692" watchObservedRunningTime="2026-09-15 21:16:43.089712129 +0000 UTC m=+43.724864243" Sep 15 21:16:43.123410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.122349 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nnvbn" podStartSLOduration=3.805543037 podStartE2EDuration="7.122331549s" totalImagesPullingTime="3.316788512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:43.121076391 +0000 UTC m=+43.756228514" watchObservedRunningTime="2026-09-15 21:16:43.122331549 +0000 UTC m=+43.757483660" Sep 15 21:16:43.986130 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.986096 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/1.log" Sep 15 21:16:43.986586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.986566 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/0.log" Sep 15 21:16:43.986661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.986605 2552 generic.go:356] "Generic (PLEG): container finished" podID="c716d930-781b-428b-b942-0129be7d46fe" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" exitCode=255 Sep 15 21:16:43.986879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.986844 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" event={"ID":"c716d930-781b-428b-b942-0129be7d46fe","Type":"ContainerDied","Data":"b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed"} Sep 15 21:16:43.987013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.986913 2552 scope.go:117] "RemoveContainer" containerID="bf1646c49c34057ae24f56cb05d1768ee8de2b5d6db20b5e10846f7e51454b6b" Sep 15 21:16:43.987481 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:43.987263 2552 scope.go:117] "RemoveContainer" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" Sep 15 21:16:43.987481 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:43.987431 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:16:44.133929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.133881 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gwdvd"] Sep 15 21:16:44.178713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.178682 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwdvd"] Sep 15 21:16:44.178910 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.178816 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.181410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.181388 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 21:16:44.181523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.181394 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 21:16:44.181816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.181798 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6jkf\"" Sep 15 21:16:44.259039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.258954 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/1168da50-436b-4731-ba66-99dfe04d4b83-kube-api-access-v4924\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.259039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.259008 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1168da50-436b-4731-ba66-99dfe04d4b83-crio-socket\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.259303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.259111 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.259303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.259136 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1168da50-436b-4731-ba66-99dfe04d4b83-data-volume\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.259303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.259176 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.359846 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.359803 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1168da50-436b-4731-ba66-99dfe04d4b83-crio-socket\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.359931 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.359965 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1168da50-436b-4731-ba66-99dfe04d4b83-data-volume\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.360010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1168da50-436b-4731-ba66-99dfe04d4b83-crio-socket\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360199 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.360063 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:44.360199 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.360106 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360199 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.360136 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.860115801 +0000 UTC m=+45.495267895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : configmap "kube-rbac-proxy" not found Sep 15 21:16:44.360356 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.360212 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.360356 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.360230 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/1168da50-436b-4731-ba66-99dfe04d4b83-kube-api-access-v4924\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.360356 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.360260 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.860243313 +0000 UTC m=+45.495395416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.360356 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.360282 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1168da50-436b-4731-ba66-99dfe04d4b83-data-volume\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.376330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.376287 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/1168da50-436b-4731-ba66-99dfe04d4b83-kube-api-access-v4924\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.865526 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.865490 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.865533 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.865572 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.865622 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865668 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865698 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865722 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:52.865707029 +0000 UTC m=+53.500859132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:44.865731 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865728 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865741 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.865732136 +0000 UTC m=+46.500884226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865757 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:52.865747565 +0000 UTC m=+53.500899657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.865780 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865806 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:52.865786059 +0000 UTC m=+53.500938160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : secret "router-metrics-certs-default" not found Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865808 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:44.866043 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.865884 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.86587278 +0000 UTC m=+46.501024883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : configmap "kube-rbac-proxy" not found Sep 15 21:16:44.990847 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.990810 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/1.log" Sep 15 21:16:44.991182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:44.991116 2552 scope.go:117] "RemoveContainer" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" Sep 15 21:16:44.991272 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:44.991255 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:16:45.168282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:45.168253 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:45.168455 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.168407 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:45.168455 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.168425 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58cb5bd455-nzfms: secret "image-registry-tls" not found Sep 15 21:16:45.168581 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.168486 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls podName:b58dc70f-4553-4fcb-9e80-346edd6e4d40 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:53.168464733 +0000 UTC m=+53.803616824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls") pod "image-registry-58cb5bd455-nzfms" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40") : secret "image-registry-tls" not found Sep 15 21:16:45.875068 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:45.875029 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:45.875335 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:45.875088 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:45.875335 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.875176 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:45.875335 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.875202 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:45.875335 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.875255 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:47.875232424 +0000 UTC m=+48.510384527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : configmap "kube-rbac-proxy" not found Sep 15 21:16:45.875335 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:45.875273 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:47.875264085 +0000 UTC m=+48.510416180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:45.995182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:45.995147 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" event={"ID":"d8afb728-416e-4d22-a63f-bb14feb28147","Type":"ContainerStarted","Data":"cce31c0f3be1f831732504d5bb5a627d6b55f7792be18b441fdc5cde42add321"} Sep 15 21:16:46.026479 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:46.026436 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vlzv" podStartSLOduration=3.046430739 podStartE2EDuration="9.026421156s" totalImagesPullingTime="5.979990417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:46.026297144 +0000 UTC m=+46.661449257" watchObservedRunningTime="2026-09-15 21:16:46.026421156 +0000 UTC m=+46.661573264" Sep 15 21:16:46.287627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:46.287559 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fx9kz_757e5f7e-9388-48af-bc43-6ed61d49a316/dns-node-resolver/0.log" Sep 15 21:16:47.394408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.394368 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:47.394914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.394435 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:16:47.394914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.394867 2552 scope.go:117] "RemoveContainer" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" Sep 15 21:16:47.395087 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:47.395066 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:16:47.512504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.512480 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hj8x9_fa5d8f7e-3092-496c-a773-1d0ef3c3295f/node-ca/0.log" Sep 15 21:16:47.892907 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.892874 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:47.893084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:47.892926 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:47.893084 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:47.893002 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:47.893084 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:47.893066 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:51.893050594 +0000 UTC m=+52.528202691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : configmap "kube-rbac-proxy" not found Sep 15 21:16:47.893249 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:47.893105 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:47.893249 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:47.893163 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:51.893145328 +0000 UTC m=+52.528297432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:48.088344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:48.088295 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fjdlx_c71e5357-0147-4721-a31c-09e2e67ce367/serve-healthcheck-canary/0.log" Sep 15 21:16:49.609348 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:49.609313 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:16:49.609732 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:49.609470 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:49.609732 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:49.609543 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls podName:9238defc-83a1-40db-9a48-7002cd3a4010 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:05.609525956 +0000 UTC m=+66.244678044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls") pod "dns-default-vwjjx" (UID: "9238defc-83a1-40db-9a48-7002cd3a4010") : secret "dns-default-metrics-tls" not found Sep 15 21:16:51.930932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:51.930893 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:51.931410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:51.930948 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:16:51.931410 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:51.931087 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:51.931410 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:51.931136 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:51.931410 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:51.931175 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.93115249 +0000 UTC m=+60.566304579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : configmap "kube-rbac-proxy" not found Sep 15 21:16:51.931410 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:51.931200 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls podName:1168da50-436b-4731-ba66-99dfe04d4b83 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.931184585 +0000 UTC m=+60.566336679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwdvd" (UID: "1168da50-436b-4731-ba66-99dfe04d4b83") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:52.938098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:52.938040 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:16:52.938577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:52.938135 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:52.938577 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:52.938192 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:52.938577 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:52.938266 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls podName:637e59d0-4000-4180-a0ca-92c166ac1061 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:08.938250631 +0000 UTC m=+69.573402724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8c4js" (UID: "637e59d0-4000-4180-a0ca-92c166ac1061") : secret "cluster-monitoring-operator-tls" not found Sep 15 21:16:52.938577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:52.938202 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:52.938577 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:52.938307 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle podName:2dedd29e-1f9c-4549-8697-44de0145c8e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:08.93829277 +0000 UTC m=+69.573444876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle") pod "router-default-5cdd487d68-4972n" (UID: "2dedd29e-1f9c-4549-8697-44de0145c8e6") : configmap references non-existent config key: service-ca.crt Sep 15 21:16:52.940465 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:52.940430 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dedd29e-1f9c-4549-8697-44de0145c8e6-metrics-certs\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:16:53.241003 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:53.240920 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:53.243228 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:53.243209 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"image-registry-58cb5bd455-nzfms\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:53.542014 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:53.541925 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:53.811588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:53.811517 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:16:53.814955 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:16:53.814926 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58dc70f_4553_4fcb_9e80_346edd6e4d40.slice/crio-f89c8e26792e6fd0df21a244ab31df2a6d3b720b0d699413384a490105063bf1 WatchSource:0}: Error finding container f89c8e26792e6fd0df21a244ab31df2a6d3b720b0d699413384a490105063bf1: Status 404 returned error can't find the container with id f89c8e26792e6fd0df21a244ab31df2a6d3b720b0d699413384a490105063bf1 Sep 15 21:16:54.020744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:54.020709 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" event={"ID":"b58dc70f-4553-4fcb-9e80-346edd6e4d40","Type":"ContainerStarted","Data":"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010"} Sep 15 21:16:54.020744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:54.020769 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" event={"ID":"b58dc70f-4553-4fcb-9e80-346edd6e4d40","Type":"ContainerStarted","Data":"f89c8e26792e6fd0df21a244ab31df2a6d3b720b0d699413384a490105063bf1"} Sep 15 21:16:54.021336 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:54.020869 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:16:54.041681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:54.041604 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" podStartSLOduration=17.041583422 podStartE2EDuration="17.041583422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:54.041454278 +0000 UTC m=+54.676606415" watchObservedRunningTime="2026-09-15 21:16:54.041583422 +0000 UTC m=+54.676735534" Sep 15 21:16:58.784048 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:58.784015 2552 scope.go:117] "RemoveContainer" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" Sep 15 21:16:59.040751 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.040679 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:16:59.041209 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.041194 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/1.log" Sep 15 21:16:59.041269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.041227 2552 generic.go:356] "Generic (PLEG): container finished" podID="c716d930-781b-428b-b942-0129be7d46fe" containerID="382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1" exitCode=255 Sep 15 21:16:59.041307 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.041279 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" event={"ID":"c716d930-781b-428b-b942-0129be7d46fe","Type":"ContainerDied","Data":"382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1"} Sep 15 21:16:59.041347 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.041324 2552 scope.go:117] "RemoveContainer" containerID="b5a9a6b224dffa17f62923110d601e68bdc491fd9885c45fc336598b23a78bed" Sep 15 21:16:59.041651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.041635 2552 scope.go:117] "RemoveContainer" containerID="382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1" Sep 15 21:16:59.041836 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:16:59.041796 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:16:59.923669 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:16:59.923641 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrpkm" Sep 15 21:17:00.000269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.000223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:17:00.000460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.000287 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:17:00.000869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.000823 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1168da50-436b-4731-ba66-99dfe04d4b83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:17:00.002750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.002729 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1168da50-436b-4731-ba66-99dfe04d4b83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwdvd\" (UID: \"1168da50-436b-4731-ba66-99dfe04d4b83\") " pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:17:00.045895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.045865 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:17:00.091717 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.091683 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6jkf\"" Sep 15 21:17:00.100044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.100022 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwdvd" Sep 15 21:17:00.396852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:00.396730 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwdvd"] Sep 15 21:17:00.399404 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:00.399377 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1168da50_436b_4731_ba66_99dfe04d4b83.slice/crio-4a7b4ef96b3a98072aae9d0e8f6a98896bbee3e8562759077d7a42225d036ad9 WatchSource:0}: Error finding container 4a7b4ef96b3a98072aae9d0e8f6a98896bbee3e8562759077d7a42225d036ad9: Status 404 returned error can't find the container with id 4a7b4ef96b3a98072aae9d0e8f6a98896bbee3e8562759077d7a42225d036ad9 Sep 15 21:17:01.051486 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:01.051451 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwdvd" event={"ID":"1168da50-436b-4731-ba66-99dfe04d4b83","Type":"ContainerStarted","Data":"28c47e819757ee6549fa7cd021e0cd72f4d949c714e9068c061a86d3a4da3ef9"} Sep 15 21:17:01.051877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:01.051537 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwdvd" event={"ID":"1168da50-436b-4731-ba66-99dfe04d4b83","Type":"ContainerStarted","Data":"4a7b4ef96b3a98072aae9d0e8f6a98896bbee3e8562759077d7a42225d036ad9"} Sep 15 21:17:02.056117 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:02.056036 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwdvd" event={"ID":"1168da50-436b-4731-ba66-99dfe04d4b83","Type":"ContainerStarted","Data":"8acfc0c71e26575b0a29f2ae96f452a8517c9d66e3770c60bfc4d4cc675fdaf5"} Sep 15 21:17:04.064756 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:04.064721 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwdvd" event={"ID":"1168da50-436b-4731-ba66-99dfe04d4b83","Type":"ContainerStarted","Data":"27e908baf038658d83544be14c491778aa6311fc4065c8fed4bded037174ca4f"} Sep 15 21:17:04.088386 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:04.088302 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gwdvd" podStartSLOduration=17.455617494 podStartE2EDuration="20.08828703s" totalImagesPullingTime="2.632669536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:04.087099581 +0000 UTC m=+64.722251692" watchObservedRunningTime="2026-09-15 21:17:04.08828703 +0000 UTC m=+64.723439141" Sep 15 21:17:05.551044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.550992 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:17:05.553901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.553882 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:17:05.563880 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.563860 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0487501f-1be4-4d51-8ba4-8314af778987-original-pull-secret\") pod \"global-pull-secret-syncer-6w6wz\" (UID: \"0487501f-1be4-4d51-8ba4-8314af778987\") " pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:17:05.604583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.604548 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w6wz" Sep 15 21:17:05.661936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.653202 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:17:05.661936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.653263 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:17:05.661936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.653299 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:17:05.661936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.656205 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmgx\" (UniqueName: \"kubernetes.io/projected/cd7ffb29-5e39-4f8b-a217-ad34ede637ce-kube-api-access-2qmgx\") pod \"network-check-target-m2l6s\" (UID: \"cd7ffb29-5e39-4f8b-a217-ad34ede637ce\") " pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:17:05.676264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.676006 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:17:05.676877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.676812 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9238defc-83a1-40db-9a48-7002cd3a4010-metrics-tls\") pod \"dns-default-vwjjx\" (UID: \"9238defc-83a1-40db-9a48-7002cd3a4010\") " pod="openshift-dns/dns-default-vwjjx" Sep 15 21:17:05.687473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.687445 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70176b6-167d-40d3-81ff-d27f61b9ad08-metrics-certs\") pod \"network-metrics-daemon-9fzbx\" (UID: \"e70176b6-167d-40d3-81ff-d27f61b9ad08\") " pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:17:05.835925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.835842 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ztstg\"" Sep 15 21:17:05.843902 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.843881 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwjjx" Sep 15 21:17:05.887299 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.887269 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6w6wz"] Sep 15 21:17:05.892773 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:05.892739 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0487501f_1be4_4d51_8ba4_8314af778987.slice/crio-2ed7a5354339ffb05980bcff2e50a5f29026910f5677407f710f3284a2c46c3b WatchSource:0}: Error finding container 2ed7a5354339ffb05980bcff2e50a5f29026910f5677407f710f3284a2c46c3b: Status 404 returned error can't find the container with id 2ed7a5354339ffb05980bcff2e50a5f29026910f5677407f710f3284a2c46c3b Sep 15 21:17:05.896258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.896236 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\"" Sep 15 21:17:05.902887 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.902865 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\"" Sep 15 21:17:05.903400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.903380 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fzbx" Sep 15 21:17:05.911895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:05.911871 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:17:06.072951 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:06.072915 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6w6wz" event={"ID":"0487501f-1be4-4d51-8ba4-8314af778987","Type":"ContainerStarted","Data":"2ed7a5354339ffb05980bcff2e50a5f29026910f5677407f710f3284a2c46c3b"} Sep 15 21:17:06.128104 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:06.128014 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vwjjx"] Sep 15 21:17:06.131623 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:06.131599 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9238defc_83a1_40db_9a48_7002cd3a4010.slice/crio-17109e8789b16777fd4b0d3be00544a8b18e43338a852bdf4395db54aafd53b7 WatchSource:0}: Error finding container 17109e8789b16777fd4b0d3be00544a8b18e43338a852bdf4395db54aafd53b7: Status 404 returned error can't find the container with id 17109e8789b16777fd4b0d3be00544a8b18e43338a852bdf4395db54aafd53b7 Sep 15 21:17:06.197821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:06.197782 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fzbx"] Sep 15 21:17:06.199987 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:06.199965 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2l6s"] Sep 15 21:17:06.201859 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:06.201820 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70176b6_167d_40d3_81ff_d27f61b9ad08.slice/crio-4fe04d449406e1c76ec18e0729b2fb3eff52327e1195a029e770f5da10a68084 WatchSource:0}: Error finding container 4fe04d449406e1c76ec18e0729b2fb3eff52327e1195a029e770f5da10a68084: Status 404 returned error can't find the container with id 4fe04d449406e1c76ec18e0729b2fb3eff52327e1195a029e770f5da10a68084 Sep 15 21:17:06.203167 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:06.203146 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7ffb29_5e39_4f8b_a217_ad34ede637ce.slice/crio-97e2108bf6da460fb191d1e8953f16f727839673a4bd321c63227c498055cb11 WatchSource:0}: Error finding container 97e2108bf6da460fb191d1e8953f16f727839673a4bd321c63227c498055cb11: Status 404 returned error can't find the container with id 97e2108bf6da460fb191d1e8953f16f727839673a4bd321c63227c498055cb11 Sep 15 21:17:07.081877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.081647 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2l6s" event={"ID":"cd7ffb29-5e39-4f8b-a217-ad34ede637ce","Type":"ContainerStarted","Data":"d8b6e4492f975d4e84e12ae8a66f91ed0c1fc8f8cdfb43fa843cf2c4122e4837"} Sep 15 21:17:07.081877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.081706 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2l6s" event={"ID":"cd7ffb29-5e39-4f8b-a217-ad34ede637ce","Type":"ContainerStarted","Data":"97e2108bf6da460fb191d1e8953f16f727839673a4bd321c63227c498055cb11"} Sep 15 21:17:07.081877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.081762 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:17:07.083436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.083322 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwjjx" event={"ID":"9238defc-83a1-40db-9a48-7002cd3a4010","Type":"ContainerStarted","Data":"17109e8789b16777fd4b0d3be00544a8b18e43338a852bdf4395db54aafd53b7"} Sep 15 21:17:07.085004 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.084934 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fzbx" event={"ID":"e70176b6-167d-40d3-81ff-d27f61b9ad08","Type":"ContainerStarted","Data":"4fe04d449406e1c76ec18e0729b2fb3eff52327e1195a029e770f5da10a68084"} Sep 15 21:17:07.101882 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.101618 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2l6s" podStartSLOduration=67.101597837 podStartE2EDuration="1m7.101597837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:07.101280418 +0000 UTC m=+67.736432530" watchObservedRunningTime="2026-09-15 21:17:07.101597837 +0000 UTC m=+67.736749950" Sep 15 21:17:07.394442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.394404 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:17:07.394627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.394473 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:17:07.394899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.394881 2552 scope.go:117] "RemoveContainer" containerID="382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1" Sep 15 21:17:07.395095 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:17:07.395065 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:17:07.858287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.858197 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd"] Sep 15 21:17:07.878737 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.878698 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:07.885068 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.885041 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 21:17:07.897047 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.897017 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn"] Sep 15 21:17:07.910239 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.910213 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 21:17:07.914892 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.914806 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:07.938011 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.937986 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 15 21:17:07.938167 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.937986 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-brkq7\"" Sep 15 21:17:07.940288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.940264 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 15 21:17:07.945300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.945276 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 21:17:07.957812 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.957745 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd"] Sep 15 21:17:07.959367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.959341 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn"] Sep 15 21:17:07.973390 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.973352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ccb9f0-4979-498e-a4ba-ac820315c5a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:07.973540 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:07.973426 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcnp\" (UniqueName: \"kubernetes.io/projected/00ccb9f0-4979-498e-a4ba-ac820315c5a7-kube-api-access-ngcnp\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.074434 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.074395 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.074630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.074461 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6jq9\" (UniqueName: \"kubernetes.io/projected/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-kube-api-access-v6jq9\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.074630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.074514 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ccb9f0-4979-498e-a4ba-ac820315c5a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.074630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.074564 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-tmp\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.074630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.074599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcnp\" (UniqueName: \"kubernetes.io/projected/00ccb9f0-4979-498e-a4ba-ac820315c5a7-kube-api-access-ngcnp\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.078044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.078014 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ccb9f0-4979-498e-a4ba-ac820315c5a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.099624 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.099593 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcnp\" (UniqueName: \"kubernetes.io/projected/00ccb9f0-4979-498e-a4ba-ac820315c5a7-kube-api-access-ngcnp\") pod \"managed-serviceaccount-addon-agent-847df6b778-84hfd\" (UID: \"00ccb9f0-4979-498e-a4ba-ac820315c5a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.171502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.171470 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:17:08.175946 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.175913 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-tmp\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.176094 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.176001 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.176094 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.176043 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6jq9\" (UniqueName: \"kubernetes.io/projected/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-kube-api-access-v6jq9\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.176532 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.176492 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-tmp\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.176685 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.176543 2552 patch_prober.go:28] interesting pod/image-registry-58cb5bd455-nzfms container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 15 21:17:08.176685 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.176625 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:17:08.178767 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.178742 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.193340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.193317 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6jq9\" (UniqueName: \"kubernetes.io/projected/f1b2a69b-5c09-40b4-ace2-2b160b28f4fc-kube-api-access-v6jq9\") pod \"klusterlet-addon-workmgr-6ff598d5fc-6h9gn\" (UID: \"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.200981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.200960 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" Sep 15 21:17:08.227268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.227249 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:08.508983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.508902 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd"] Sep 15 21:17:08.543575 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:08.543532 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00ccb9f0_4979_498e_a4ba_ac820315c5a7.slice/crio-c13fd70f9e9b62ed0a8eb87390416071b37707f1effbb1787403f4c6eb146d4a WatchSource:0}: Error finding container c13fd70f9e9b62ed0a8eb87390416071b37707f1effbb1787403f4c6eb146d4a: Status 404 returned error can't find the container with id c13fd70f9e9b62ed0a8eb87390416071b37707f1effbb1787403f4c6eb146d4a Sep 15 21:17:08.558313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.558283 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn"] Sep 15 21:17:08.577597 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:08.577569 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b2a69b_5c09_40b4_ace2_2b160b28f4fc.slice/crio-c874018161e598e6cabce28682adbf5e6d931907a8ec9ca0f7b3cee6999199c5 WatchSource:0}: Error finding container c874018161e598e6cabce28682adbf5e6d931907a8ec9ca0f7b3cee6999199c5: Status 404 returned error can't find the container with id c874018161e598e6cabce28682adbf5e6d931907a8ec9ca0f7b3cee6999199c5 Sep 15 21:17:08.982556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.982521 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:08.982719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.982572 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:17:08.983377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.983348 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dedd29e-1f9c-4549-8697-44de0145c8e6-service-ca-bundle\") pod \"router-default-5cdd487d68-4972n\" (UID: \"2dedd29e-1f9c-4549-8697-44de0145c8e6\") " pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:08.985083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:08.985054 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/637e59d0-4000-4180-a0ca-92c166ac1061-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8c4js\" (UID: \"637e59d0-4000-4180-a0ca-92c166ac1061\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:17:09.095579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.095541 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwjjx" event={"ID":"9238defc-83a1-40db-9a48-7002cd3a4010","Type":"ContainerStarted","Data":"e806d4aaf61acce2b0f8cdab255a0b83945f63ef31713ae3709c3f5eefb6b957"} Sep 15 21:17:09.097002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.096973 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" event={"ID":"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc","Type":"ContainerStarted","Data":"c874018161e598e6cabce28682adbf5e6d931907a8ec9ca0f7b3cee6999199c5"} Sep 15 21:17:09.099203 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.099174 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fzbx" event={"ID":"e70176b6-167d-40d3-81ff-d27f61b9ad08","Type":"ContainerStarted","Data":"81ccb9951ef1bfb639adc8b55bb3503946ffc1388a0a030de4b33913ce6fef4b"} Sep 15 21:17:09.099315 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.099225 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fzbx" event={"ID":"e70176b6-167d-40d3-81ff-d27f61b9ad08","Type":"ContainerStarted","Data":"be60f29c178c8d9d657d6dcb2c752d477b40d5f65c400078c74f89bd3ef4785c"} Sep 15 21:17:09.100383 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.100360 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" event={"ID":"00ccb9f0-4979-498e-a4ba-ac820315c5a7","Type":"ContainerStarted","Data":"c13fd70f9e9b62ed0a8eb87390416071b37707f1effbb1787403f4c6eb146d4a"} Sep 15 21:17:09.131861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.131792 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9fzbx" podStartSLOduration=67.132185872 podStartE2EDuration="1m9.131776689s" totalImagesPullingTime="1.999590817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:09.1315347 +0000 UTC m=+69.766686813" watchObservedRunningTime="2026-09-15 21:17:09.131776689 +0000 UTC m=+69.766928847" Sep 15 21:17:09.184648 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.184615 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mgbts\"" Sep 15 21:17:09.191811 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.191781 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" Sep 15 21:17:09.205408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.205377 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qtspj\"" Sep 15 21:17:09.214064 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:09.214024 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:10.398927 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:10.398822 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js"] Sep 15 21:17:10.403000 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:10.402967 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637e59d0_4000_4180_a0ca_92c166ac1061.slice/crio-5d8d9078776da484fa4422c615074dbd08524ef947c91a21ce9380bc1686970d WatchSource:0}: Error finding container 5d8d9078776da484fa4422c615074dbd08524ef947c91a21ce9380bc1686970d: Status 404 returned error can't find the container with id 5d8d9078776da484fa4422c615074dbd08524ef947c91a21ce9380bc1686970d Sep 15 21:17:10.403502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:10.403475 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cdd487d68-4972n"] Sep 15 21:17:10.407296 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:10.407262 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dedd29e_1f9c_4549_8697_44de0145c8e6.slice/crio-7edd2f88daecda069ab4334ce8320fb54e1fe83b61eb061a872c5a39fbd65c6d WatchSource:0}: Error finding container 7edd2f88daecda069ab4334ce8320fb54e1fe83b61eb061a872c5a39fbd65c6d: Status 404 returned error can't find the container with id 7edd2f88daecda069ab4334ce8320fb54e1fe83b61eb061a872c5a39fbd65c6d Sep 15 21:17:11.111009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.110675 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6w6wz" event={"ID":"0487501f-1be4-4d51-8ba4-8314af778987","Type":"ContainerStarted","Data":"b353f9ebab557ad1f659b92f7b19b564ca4d39b6dc46c5b0141468363e463d75"} Sep 15 21:17:11.112523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.112493 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cdd487d68-4972n" event={"ID":"2dedd29e-1f9c-4549-8697-44de0145c8e6","Type":"ContainerStarted","Data":"a2b9a270d906393dc3bd33fea9e307525fee6daffef3dbb067d35f5534c64b26"} Sep 15 21:17:11.112687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.112547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cdd487d68-4972n" event={"ID":"2dedd29e-1f9c-4549-8697-44de0145c8e6","Type":"ContainerStarted","Data":"7edd2f88daecda069ab4334ce8320fb54e1fe83b61eb061a872c5a39fbd65c6d"} Sep 15 21:17:11.115114 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.115083 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwjjx" event={"ID":"9238defc-83a1-40db-9a48-7002cd3a4010","Type":"ContainerStarted","Data":"7834d9e440aa40a0e3eee1987687532f5321d5c9f5577a9f9630a3749fc91db9"} Sep 15 21:17:11.115255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.115160 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vwjjx" Sep 15 21:17:11.116638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.116602 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" event={"ID":"637e59d0-4000-4180-a0ca-92c166ac1061","Type":"ContainerStarted","Data":"5d8d9078776da484fa4422c615074dbd08524ef947c91a21ce9380bc1686970d"} Sep 15 21:17:11.129066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.128913 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6w6wz" podStartSLOduration=66.761500331 podStartE2EDuration="1m11.128885889s" totalImagesPullingTime="4.367385558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:11.127277709 +0000 UTC m=+71.762429844" watchObservedRunningTime="2026-09-15 21:17:11.128885889 +0000 UTC m=+71.764038003" Sep 15 21:17:11.147598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.147538 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vwjjx" podStartSLOduration=35.702968574 podStartE2EDuration="38.147517407s" totalImagesPullingTime="2.444548833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:11.146510603 +0000 UTC m=+71.781662714" watchObservedRunningTime="2026-09-15 21:17:11.147517407 +0000 UTC m=+71.782669518" Sep 15 21:17:11.166763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.166715 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cdd487d68-4972n" podStartSLOduration=34.166697105 podStartE2EDuration="34.166697105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:11.165933339 +0000 UTC m=+71.801085447" watchObservedRunningTime="2026-09-15 21:17:11.166697105 +0000 UTC m=+71.801849217" Sep 15 21:17:11.214901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.214868 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:11.218016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:11.217991 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:12.123408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:12.123319 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:12.124255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:12.124065 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cdd487d68-4972n" Sep 15 21:17:16.137640 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.137607 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" event={"ID":"00ccb9f0-4979-498e-a4ba-ac820315c5a7","Type":"ContainerStarted","Data":"327071e0b38f52d42cc3e3fbb185ba91fb35e49943e337a0680a4cec78cd79b9"} Sep 15 21:17:16.139020 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.138995 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" event={"ID":"f1b2a69b-5c09-40b4-ace2-2b160b28f4fc","Type":"ContainerStarted","Data":"6207ca303785525e4ab29b5bf05c7ba87c2f753aaf42a1e5c52d4af789fdf571"} Sep 15 21:17:16.139141 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.139045 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:16.140422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.140399 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" event={"ID":"637e59d0-4000-4180-a0ca-92c166ac1061","Type":"ContainerStarted","Data":"045cc55ee6f1aee951f909d1791801716232b22c4944c10dd3c10ef7f7963990"} Sep 15 21:17:16.141000 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.140980 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" Sep 15 21:17:16.153215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.153169 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847df6b778-84hfd" podStartSLOduration=2.5964991 podStartE2EDuration="9.15315649s" totalImagesPullingTime="6.55665739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:16.152322337 +0000 UTC m=+76.787474450" watchObservedRunningTime="2026-09-15 21:17:16.15315649 +0000 UTC m=+76.788308598" Sep 15 21:17:16.166981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.166927 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8c4js" podStartSLOduration=34.460984883 podStartE2EDuration="39.16690856s" totalImagesPullingTime="4.705923677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:16.166795932 +0000 UTC m=+76.801948043" watchObservedRunningTime="2026-09-15 21:17:16.16690856 +0000 UTC m=+76.802060673" Sep 15 21:17:16.182235 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:16.182192 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ff598d5fc-6h9gn" podStartSLOduration=2.652783125 podStartE2EDuration="9.182179237s" totalImagesPullingTime="6.529396112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:16.181044725 +0000 UTC m=+76.816196840" watchObservedRunningTime="2026-09-15 21:17:16.182179237 +0000 UTC m=+76.817331348" Sep 15 21:17:18.125944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:18.125917 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vwjjx" Sep 15 21:17:18.176603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:18.176576 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:17:18.784890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:18.784863 2552 scope.go:117] "RemoveContainer" containerID="382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1" Sep 15 21:17:18.785086 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:17:18.785036 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-2ss7v_openshift-console-operator(c716d930-781b-428b-b942-0129be7d46fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podUID="c716d930-781b-428b-b942-0129be7d46fe" Sep 15 21:17:24.149748 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.149624 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gnrqq"] Sep 15 21:17:24.152542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.152523 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.155036 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.155006 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 21:17:24.155159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.155054 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 21:17:24.155159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.155068 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 21:17:24.155159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.155019 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 21:17:24.155885 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.155867 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hp94g\"" Sep 15 21:17:24.301613 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301577 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-metrics-client-ca\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301613 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301618 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301643 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301659 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkc2\" (UniqueName: \"kubernetes.io/projected/a868f3da-5875-494a-9f77-78e52ac70823-kube-api-access-ggkc2\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301697 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-root\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301727 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-tls\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301746 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-wtmp\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301770 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-sys\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.301866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.301819 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-textfile\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402653 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402577 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-sys\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402656 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-textfile\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402697 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-sys\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402724 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-metrics-client-ca\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402746 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402787 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.402862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402812 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkc2\" (UniqueName: \"kubernetes.io/projected/a868f3da-5875-494a-9f77-78e52ac70823-kube-api-access-ggkc2\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402871 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-root\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402910 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-tls\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.402934 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-wtmp\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.403033 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-root\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.403076 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-textfile\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.403171 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-wtmp\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403618 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.403595 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-metrics-client-ca\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.403738 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.403722 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.405805 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.405781 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.405805 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.405800 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a868f3da-5875-494a-9f77-78e52ac70823-node-exporter-tls\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.417536 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.417511 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkc2\" (UniqueName: \"kubernetes.io/projected/a868f3da-5875-494a-9f77-78e52ac70823-kube-api-access-ggkc2\") pod \"node-exporter-gnrqq\" (UID: \"a868f3da-5875-494a-9f77-78e52ac70823\") " pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.461420 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:24.461387 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gnrqq" Sep 15 21:17:24.471302 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:24.471272 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda868f3da_5875_494a_9f77_78e52ac70823.slice/crio-bdd58890cf4038b981f5a72d74cea9e08d9f1ab56b14e894656fcc1e5007938e WatchSource:0}: Error finding container bdd58890cf4038b981f5a72d74cea9e08d9f1ab56b14e894656fcc1e5007938e: Status 404 returned error can't find the container with id bdd58890cf4038b981f5a72d74cea9e08d9f1ab56b14e894656fcc1e5007938e Sep 15 21:17:25.174969 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:25.174931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrqq" event={"ID":"a868f3da-5875-494a-9f77-78e52ac70823","Type":"ContainerStarted","Data":"bdd58890cf4038b981f5a72d74cea9e08d9f1ab56b14e894656fcc1e5007938e"} Sep 15 21:17:26.184298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:26.184257 2552 generic.go:356] "Generic (PLEG): container finished" podID="a868f3da-5875-494a-9f77-78e52ac70823" containerID="b2030b5deaeb7fb98d21a3c04df7d88170669f1d6680800317e8b903cbcbe357" exitCode=0 Sep 15 21:17:26.184663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:26.184338 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrqq" event={"ID":"a868f3da-5875-494a-9f77-78e52ac70823","Type":"ContainerDied","Data":"b2030b5deaeb7fb98d21a3c04df7d88170669f1d6680800317e8b903cbcbe357"} Sep 15 21:17:27.192761 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:27.192727 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrqq" event={"ID":"a868f3da-5875-494a-9f77-78e52ac70823","Type":"ContainerStarted","Data":"ef53f5d3b7b7c1aa8133a31960d0aa92d31bf23517a92ac6ea787bb3e3dd8368"} Sep 15 21:17:27.193122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:27.192783 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrqq" event={"ID":"a868f3da-5875-494a-9f77-78e52ac70823","Type":"ContainerStarted","Data":"193714d0b88a5bc9d9866b7723fdbf965faa3af98fceea232a08e1eb7f471d26"} Sep 15 21:17:27.218914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:27.218755 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gnrqq" podStartSLOduration=1.962989729 podStartE2EDuration="3.218739915s" totalImagesPullingTime="1.192820732s" totalInitContainerRuntime="62.929454ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:27.217698721 +0000 UTC m=+87.852850832" watchObservedRunningTime="2026-09-15 21:17:27.218739915 +0000 UTC m=+87.853892025" Sep 15 21:17:29.786103 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:29.786073 2552 scope.go:117] "RemoveContainer" containerID="382b0f96feea51dd5c41fba9822f8b78b953ed7fd0780c7a87ef383bf4869cc1" Sep 15 21:17:30.204520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.204490 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:17:30.204714 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.204544 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" event={"ID":"c716d930-781b-428b-b942-0129be7d46fe","Type":"ContainerStarted","Data":"56380a6c2e3318638897078d8a1d671a63a331e6b9bd413347fae8120ed70594"} Sep 15 21:17:30.204879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.204859 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:17:30.220933 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.220868 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" podStartSLOduration=49.872470114 podStartE2EDuration="53.220823318s" totalImagesPullingTime="3.348353204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:30.220380093 +0000 UTC m=+90.855532197" watchObservedRunningTime="2026-09-15 21:17:30.220823318 +0000 UTC m=+90.855975431" Sep 15 21:17:30.293887 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.293858 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-2ss7v" Sep 15 21:17:30.511334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.511241 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-c59gt"] Sep 15 21:17:30.513261 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.513243 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:30.516627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.516602 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 15 21:17:30.516732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.516664 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 15 21:17:30.518419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.518402 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k6t9w\"" Sep 15 21:17:30.537249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.536631 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c59gt"] Sep 15 21:17:30.547176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.547141 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhq7\" (UniqueName: \"kubernetes.io/projected/cd41948e-b0c0-496d-83b3-4d2d492e3e86-kube-api-access-skhq7\") pod \"downloads-76f99584f4-c59gt\" (UID: \"cd41948e-b0c0-496d-83b3-4d2d492e3e86\") " pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:30.647885 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.647857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhq7\" (UniqueName: \"kubernetes.io/projected/cd41948e-b0c0-496d-83b3-4d2d492e3e86-kube-api-access-skhq7\") pod \"downloads-76f99584f4-c59gt\" (UID: \"cd41948e-b0c0-496d-83b3-4d2d492e3e86\") " pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:30.663934 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.663904 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhq7\" (UniqueName: \"kubernetes.io/projected/cd41948e-b0c0-496d-83b3-4d2d492e3e86-kube-api-access-skhq7\") pod \"downloads-76f99584f4-c59gt\" (UID: \"cd41948e-b0c0-496d-83b3-4d2d492e3e86\") " pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:30.822353 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:30.822271 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:31.094105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:31.094024 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c59gt"] Sep 15 21:17:31.097756 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:31.097726 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd41948e_b0c0_496d_83b3_4d2d492e3e86.slice/crio-3f328f3d7a95e4641e89ace4f166354fbb8402d7003d80d7c8e90d324e13b4c3 WatchSource:0}: Error finding container 3f328f3d7a95e4641e89ace4f166354fbb8402d7003d80d7c8e90d324e13b4c3: Status 404 returned error can't find the container with id 3f328f3d7a95e4641e89ace4f166354fbb8402d7003d80d7c8e90d324e13b4c3 Sep 15 21:17:31.208699 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:31.208665 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c59gt" event={"ID":"cd41948e-b0c0-496d-83b3-4d2d492e3e86","Type":"ContainerStarted","Data":"3f328f3d7a95e4641e89ace4f166354fbb8402d7003d80d7c8e90d324e13b4c3"} Sep 15 21:17:33.198446 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.198395 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerName="registry" containerID="cri-o://ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010" gracePeriod=30 Sep 15 21:17:33.447423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.447395 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:17:33.466743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466677 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.466743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466725 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.466948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466744 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.466948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466777 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.466948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466794 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vnv9\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.466948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.466839 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.467288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.467240 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:17:33.467409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.467298 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.467409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.467336 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates\") pod \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\" (UID: \"b58dc70f-4553-4fcb-9e80-346edd6e4d40\") " Sep 15 21:17:33.467579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.467552 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-trusted-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.468042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.468014 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:17:33.472118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.472073 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:33.472443 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.472409 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:33.477049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.477029 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9" (OuterVolumeSpecName: "kube-api-access-8vnv9") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "kube-api-access-8vnv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:33.477143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.477050 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:33.477194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.477141 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:33.477359 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.477338 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b58dc70f-4553-4fcb-9e80-346edd6e4d40" (UID: "b58dc70f-4553-4fcb-9e80-346edd6e4d40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:17:33.568613 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568580 2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-installation-pull-secrets\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568613 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568611 2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-bound-sa-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568623 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vnv9\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-kube-api-access-8vnv9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568635 2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568646 2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b58dc70f-4553-4fcb-9e80-346edd6e4d40-registry-certificates\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568657 2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b58dc70f-4553-4fcb-9e80-346edd6e4d40-ca-trust-extracted\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:33.568804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:33.568669 2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b58dc70f-4553-4fcb-9e80-346edd6e4d40-image-registry-private-configuration\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:17:34.221936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.221897 2552 generic.go:356] "Generic (PLEG): container finished" podID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerID="ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010" exitCode=0 Sep 15 21:17:34.222406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.221964 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" Sep 15 21:17:34.222406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.221991 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" event={"ID":"b58dc70f-4553-4fcb-9e80-346edd6e4d40","Type":"ContainerDied","Data":"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010"} Sep 15 21:17:34.222406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.222047 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cb5bd455-nzfms" event={"ID":"b58dc70f-4553-4fcb-9e80-346edd6e4d40","Type":"ContainerDied","Data":"f89c8e26792e6fd0df21a244ab31df2a6d3b720b0d699413384a490105063bf1"} Sep 15 21:17:34.222406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.222071 2552 scope.go:117] "RemoveContainer" containerID="ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010" Sep 15 21:17:34.236772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.236742 2552 scope.go:117] "RemoveContainer" containerID="ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010" Sep 15 21:17:34.237114 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:17:34.237081 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010\": container with ID starting with ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010 not found: ID does not exist" containerID="ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010" Sep 15 21:17:34.237234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.237121 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010"} err="failed to get container status \"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010\": rpc error: code = NotFound desc = could not find container \"ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010\": container with ID starting with ec3651f009746f32b164ceeaf5256f4bda5e2ee6d536000a068547db155be010 not found: ID does not exist" Sep 15 21:17:34.240101 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.240074 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:17:34.244575 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:34.244554 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58cb5bd455-nzfms"] Sep 15 21:17:35.789751 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:35.789712 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" path="/var/lib/kubelet/pods/b58dc70f-4553-4fcb-9e80-346edd6e4d40/volumes" Sep 15 21:17:36.381452 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.381416 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:17:36.381671 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.381660 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerName="registry" Sep 15 21:17:36.381740 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.381673 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerName="registry" Sep 15 21:17:36.381773 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.381741 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58dc70f-4553-4fcb-9e80-346edd6e4d40" containerName="registry" Sep 15 21:17:36.386439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.386411 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.390422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390397 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 15 21:17:36.390559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390461 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t4cvn\"" Sep 15 21:17:36.390559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390552 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 15 21:17:36.390676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390641 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 15 21:17:36.390732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390693 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 15 21:17:36.390732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.390699 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 15 21:17:36.395419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.395399 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:17:36.492192 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492161 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.492365 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492200 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.492365 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492233 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nsd\" (UniqueName: \"kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.492365 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492332 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.492497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492374 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.492497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.492408 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593007 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593046 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593246 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593314 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nsd\" (UniqueName: \"kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593341 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593428 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593394 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593697 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593654 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593872 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.593962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.593952 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.595545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.595527 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.595641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.595573 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.615563 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.615539 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nsd\" (UniqueName: \"kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd\") pod \"console-7d5df64744-mc48b\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.695958 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.695879 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:36.978061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:36.977982 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:17:36.981733 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:36.981704 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f0b95d_f453_420d_9819_5b47149fa786.slice/crio-e22762814bb57f4a5106af7d88efa775e9e976f445fafbb5ed3a718480aaa4ef WatchSource:0}: Error finding container e22762814bb57f4a5106af7d88efa775e9e976f445fafbb5ed3a718480aaa4ef: Status 404 returned error can't find the container with id e22762814bb57f4a5106af7d88efa775e9e976f445fafbb5ed3a718480aaa4ef Sep 15 21:17:37.237042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:37.236961 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5df64744-mc48b" event={"ID":"d4f0b95d-f453-420d-9819-5b47149fa786","Type":"ContainerStarted","Data":"e22762814bb57f4a5106af7d88efa775e9e976f445fafbb5ed3a718480aaa4ef"} Sep 15 21:17:38.093165 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:38.093133 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2l6s" Sep 15 21:17:41.258235 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:41.258177 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5df64744-mc48b" event={"ID":"d4f0b95d-f453-420d-9819-5b47149fa786","Type":"ContainerStarted","Data":"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07"} Sep 15 21:17:41.277491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:41.277441 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d5df64744-mc48b" podStartSLOduration=1.174567605 podStartE2EDuration="5.277423139s" totalImagesPullingTime="4.102855534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:41.276648183 +0000 UTC m=+101.911800314" watchObservedRunningTime="2026-09-15 21:17:41.277423139 +0000 UTC m=+101.912575250" Sep 15 21:17:44.582088 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.582046 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:17:44.624258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.624206 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:17:44.624434 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.624360 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.633608 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.633472 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 15 21:17:44.665351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665314 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggms\" (UniqueName: \"kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665364 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665485 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665535 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665585 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.665681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.665638 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.766845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.766802 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767020 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.766869 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767020 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.766903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767020 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.766948 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767020 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.767001 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.767033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggms\" (UniqueName: \"kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.767064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.767664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.767635 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.768026 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.767998 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.768128 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.768031 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.768411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.768390 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.769757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.769735 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.769892 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.769862 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.776629 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.776605 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggms\" (UniqueName: \"kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms\") pod \"console-7dc95b7fcc-5hcsj\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:44.937606 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:44.937571 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:46.696178 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:46.696145 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:46.696619 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:46.696214 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:46.702706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:46.702679 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:47.285694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:47.285668 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:17:48.642412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:48.642355 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:17:48.645317 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:17:48.645292 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd76bd16_17a2_4d46_bbd0_b23febc84220.slice/crio-95df6c548035ac8bbfa1a812b5a2fb7b7de166061cd5102eaaf3a33478137d55 WatchSource:0}: Error finding container 95df6c548035ac8bbfa1a812b5a2fb7b7de166061cd5102eaaf3a33478137d55: Status 404 returned error can't find the container with id 95df6c548035ac8bbfa1a812b5a2fb7b7de166061cd5102eaaf3a33478137d55 Sep 15 21:17:49.293040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.292993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c59gt" event={"ID":"cd41948e-b0c0-496d-83b3-4d2d492e3e86","Type":"ContainerStarted","Data":"d69dde84215237e8ac03a15ac5a827fcc664a2a7c2d04ed13256e73204b20a58"} Sep 15 21:17:49.293238 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.293130 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:49.294994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.294963 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc95b7fcc-5hcsj" event={"ID":"bd76bd16-17a2-4d46-bbd0-b23febc84220","Type":"ContainerStarted","Data":"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6"} Sep 15 21:17:49.295125 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.295006 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc95b7fcc-5hcsj" event={"ID":"bd76bd16-17a2-4d46-bbd0-b23febc84220","Type":"ContainerStarted","Data":"95df6c548035ac8bbfa1a812b5a2fb7b7de166061cd5102eaaf3a33478137d55"} Sep 15 21:17:49.304300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.304274 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-c59gt" Sep 15 21:17:49.310986 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.310940 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-c59gt" podStartSLOduration=1.799458083 podStartE2EDuration="19.310921788s" totalImagesPullingTime="17.511463705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:49.309735773 +0000 UTC m=+109.944887891" watchObservedRunningTime="2026-09-15 21:17:49.310921788 +0000 UTC m=+109.946073902" Sep 15 21:17:49.349263 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:49.349194 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc95b7fcc-5hcsj" podStartSLOduration=5.349174011 podStartE2EDuration="5.349174011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:49.348203953 +0000 UTC m=+109.983356065" watchObservedRunningTime="2026-09-15 21:17:49.349174011 +0000 UTC m=+109.984326124" Sep 15 21:17:54.938321 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:54.938282 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:54.938936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:54.938347 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:54.943866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:54.943838 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:55.331782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:55.331690 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:17:55.379047 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:17:55.379012 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:18:14.398111 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:14.398075 2552 generic.go:356] "Generic (PLEG): container finished" podID="bd64bcee-baf1-4cf4-93fd-4c0a041dd634" containerID="cb872878e96d06df7c721ba7efe21b33fa06e80b0cab4d0a469048434cca47d4" exitCode=0 Sep 15 21:18:14.398611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:14.398152 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-b24hp" event={"ID":"bd64bcee-baf1-4cf4-93fd-4c0a041dd634","Type":"ContainerDied","Data":"cb872878e96d06df7c721ba7efe21b33fa06e80b0cab4d0a469048434cca47d4"} Sep 15 21:18:14.398611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:14.398592 2552 scope.go:117] "RemoveContainer" containerID="cb872878e96d06df7c721ba7efe21b33fa06e80b0cab4d0a469048434cca47d4" Sep 15 21:18:15.403965 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:15.403925 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-b24hp" event={"ID":"bd64bcee-baf1-4cf4-93fd-4c0a041dd634","Type":"ContainerStarted","Data":"775b4e6e104d6e3e8f6e2f3d33916998984ae9d8c9ae8a4cc5d7dca9d222cf33"} Sep 15 21:18:20.403872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.403788 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d5df64744-mc48b" podUID="d4f0b95d-f453-420d-9819-5b47149fa786" containerName="console" containerID="cri-o://896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07" gracePeriod=15 Sep 15 21:18:20.681176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.681151 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:18:20.772649 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772622 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.772790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772662 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.772790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772682 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.772790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772718 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.772790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772767 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.773374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.772795 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5nsd\" (UniqueName: \"kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd\") pod \"d4f0b95d-f453-420d-9819-5b47149fa786\" (UID: \"d4f0b95d-f453-420d-9819-5b47149fa786\") " Sep 15 21:18:20.773374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.773184 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config" (OuterVolumeSpecName: "console-config") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:18:20.773374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.773226 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:18:20.773374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.773271 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:18:20.774925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.774895 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:18:20.775021 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.774947 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd" (OuterVolumeSpecName: "kube-api-access-t5nsd") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "kube-api-access-t5nsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:18:20.775021 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.774963 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4f0b95d-f453-420d-9819-5b47149fa786" (UID: "d4f0b95d-f453-420d-9819-5b47149fa786"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:18:20.874354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874320 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-console-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:20.874505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874375 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nsd\" (UniqueName: \"kubernetes.io/projected/d4f0b95d-f453-420d-9819-5b47149fa786-kube-api-access-t5nsd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:20.874578 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874561 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-oauth-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:20.874645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874609 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:20.874645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874628 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f0b95d-f453-420d-9819-5b47149fa786-console-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:20.874721 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:20.874653 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f0b95d-f453-420d-9819-5b47149fa786-oauth-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:18:21.426396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.426364 2552 generic.go:356] "Generic (PLEG): container finished" podID="d4f0b95d-f453-420d-9819-5b47149fa786" containerID="896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07" exitCode=0 Sep 15 21:18:21.426396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.426401 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5df64744-mc48b" event={"ID":"d4f0b95d-f453-420d-9819-5b47149fa786","Type":"ContainerDied","Data":"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07"} Sep 15 21:18:21.426822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.426413 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5df64744-mc48b" Sep 15 21:18:21.426822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.426435 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5df64744-mc48b" event={"ID":"d4f0b95d-f453-420d-9819-5b47149fa786","Type":"ContainerDied","Data":"e22762814bb57f4a5106af7d88efa775e9e976f445fafbb5ed3a718480aaa4ef"} Sep 15 21:18:21.426822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.426453 2552 scope.go:117] "RemoveContainer" containerID="896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07" Sep 15 21:18:21.434359 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.434345 2552 scope.go:117] "RemoveContainer" containerID="896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07" Sep 15 21:18:21.434615 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:18:21.434600 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07\": container with ID starting with 896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07 not found: ID does not exist" containerID="896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07" Sep 15 21:18:21.434655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.434623 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07"} err="failed to get container status \"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07\": rpc error: code = NotFound desc = could not find container \"896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07\": container with ID starting with 896c80f2eac669e09349b58a710a4f5b8fdefd8ad1acff2bfd886a5194004a07 not found: ID does not exist" Sep 15 21:18:21.446561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.446533 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:18:21.451269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.451248 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d5df64744-mc48b"] Sep 15 21:18:21.789558 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:21.789486 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f0b95d-f453-420d-9819-5b47149fa786" path="/var/lib/kubelet/pods/d4f0b95d-f453-420d-9819-5b47149fa786/volumes" Sep 15 21:18:52.028250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.028217 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:18:52.028732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.028479 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f0b95d-f453-420d-9819-5b47149fa786" containerName="console" Sep 15 21:18:52.028732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.028491 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f0b95d-f453-420d-9819-5b47149fa786" containerName="console" Sep 15 21:18:52.028732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.028556 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f0b95d-f453-420d-9819-5b47149fa786" containerName="console" Sep 15 21:18:52.030548 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.030527 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.047630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.047600 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:18:52.105528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105496 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9459\" (UniqueName: \"kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105572 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105623 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105671 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.105881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.105689 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206497 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9459\" (UniqueName: \"kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206544 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206581 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206601 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206616 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206636 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.206877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.206727 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.207527 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.207497 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.207663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.207571 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.207663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.207587 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.207786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.207674 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.209133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.209114 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.209231 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.209216 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.214186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.214155 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9459\" (UniqueName: \"kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459\") pod \"console-6b44dc475c-mj4d5\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.341342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.341244 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:18:52.615189 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:52.615112 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:18:52.618366 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:18:52.618339 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2d4acc_facd_4aa2_8b92_94c778e0139e.slice/crio-9c40f7454aa36ff4bda15c4e0da281970cc89d8115213e1c9f29111f4668d2f8 WatchSource:0}: Error finding container 9c40f7454aa36ff4bda15c4e0da281970cc89d8115213e1c9f29111f4668d2f8: Status 404 returned error can't find the container with id 9c40f7454aa36ff4bda15c4e0da281970cc89d8115213e1c9f29111f4668d2f8 Sep 15 21:18:53.547559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:53.547523 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b44dc475c-mj4d5" event={"ID":"1e2d4acc-facd-4aa2-8b92-94c778e0139e","Type":"ContainerStarted","Data":"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9"} Sep 15 21:18:53.547954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:53.547570 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b44dc475c-mj4d5" event={"ID":"1e2d4acc-facd-4aa2-8b92-94c778e0139e","Type":"ContainerStarted","Data":"9c40f7454aa36ff4bda15c4e0da281970cc89d8115213e1c9f29111f4668d2f8"} Sep 15 21:18:53.565663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:18:53.565612 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b44dc475c-mj4d5" podStartSLOduration=1.565596127 podStartE2EDuration="1.565596127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:53.565223785 +0000 UTC m=+174.200375894" watchObservedRunningTime="2026-09-15 21:18:53.565596127 +0000 UTC m=+174.200748238" Sep 15 21:19:02.342380 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:02.342346 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:19:02.342800 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:02.342436 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:19:02.347255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:02.347231 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:19:02.583811 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:02.583783 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:19:02.634645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:02.634611 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:19:27.659651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.659605 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dc95b7fcc-5hcsj" podUID="bd76bd16-17a2-4d46-bbd0-b23febc84220" containerName="console" containerID="cri-o://ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6" gracePeriod=15 Sep 15 21:19:27.892325 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.892302 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:19:27.990926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.990838 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.990926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.990904 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.990926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.990923 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.991194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.990963 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.991194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991013 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.991194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991040 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ggms\" (UniqueName: \"kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.991194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991071 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config\") pod \"bd76bd16-17a2-4d46-bbd0-b23febc84220\" (UID: \"bd76bd16-17a2-4d46-bbd0-b23febc84220\") " Sep 15 21:19:27.991397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991345 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:19:27.991397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991363 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:19:27.991558 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991504 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:19:27.991631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.991564 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config" (OuterVolumeSpecName: "console-config") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:19:27.993234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.993185 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms" (OuterVolumeSpecName: "kube-api-access-6ggms") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "kube-api-access-6ggms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:19:27.993482 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.993455 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:19:27.993574 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:27.993475 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd76bd16-17a2-4d46-bbd0-b23febc84220" (UID: "bd76bd16-17a2-4d46-bbd0-b23febc84220"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:19:28.092126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092092 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092121 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092131 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd76bd16-17a2-4d46-bbd0-b23febc84220-console-oauth-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092141 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092149 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-trusted-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092157 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd76bd16-17a2-4d46-bbd0-b23febc84220-oauth-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.092330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.092166 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggms\" (UniqueName: \"kubernetes.io/projected/bd76bd16-17a2-4d46-bbd0-b23febc84220-kube-api-access-6ggms\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:19:28.677255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.677223 2552 generic.go:356] "Generic (PLEG): container finished" podID="bd76bd16-17a2-4d46-bbd0-b23febc84220" containerID="ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6" exitCode=0 Sep 15 21:19:28.677622 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.677268 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc95b7fcc-5hcsj" Sep 15 21:19:28.677622 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.677295 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc95b7fcc-5hcsj" event={"ID":"bd76bd16-17a2-4d46-bbd0-b23febc84220","Type":"ContainerDied","Data":"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6"} Sep 15 21:19:28.677622 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.677335 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc95b7fcc-5hcsj" event={"ID":"bd76bd16-17a2-4d46-bbd0-b23febc84220","Type":"ContainerDied","Data":"95df6c548035ac8bbfa1a812b5a2fb7b7de166061cd5102eaaf3a33478137d55"} Sep 15 21:19:28.677622 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.677356 2552 scope.go:117] "RemoveContainer" containerID="ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6" Sep 15 21:19:28.685872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.685852 2552 scope.go:117] "RemoveContainer" containerID="ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6" Sep 15 21:19:28.686140 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:19:28.686121 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6\": container with ID starting with ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6 not found: ID does not exist" containerID="ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6" Sep 15 21:19:28.686185 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.686148 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6"} err="failed to get container status \"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6\": rpc error: code = NotFound desc = could not find container \"ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6\": container with ID starting with ce8ebf71ca9e66870a0089cfff9f906aaba9094ebb0f4135b8a2e4d61db280f6 not found: ID does not exist" Sep 15 21:19:28.697306 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.697285 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:19:28.700823 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:28.700802 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc95b7fcc-5hcsj"] Sep 15 21:19:29.789072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:19:29.789037 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd76bd16-17a2-4d46-bbd0-b23febc84220" path="/var/lib/kubelet/pods/bd76bd16-17a2-4d46-bbd0-b23febc84220/volumes" Sep 15 21:20:15.362933 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.362899 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:20:15.363362 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.363249 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd76bd16-17a2-4d46-bbd0-b23febc84220" containerName="console" Sep 15 21:20:15.363362 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.363263 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd76bd16-17a2-4d46-bbd0-b23febc84220" containerName="console" Sep 15 21:20:15.363362 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.363316 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd76bd16-17a2-4d46-bbd0-b23febc84220" containerName="console" Sep 15 21:20:15.365954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.365934 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.377862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.377816 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:20:15.431184 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431146 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431184 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431193 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431210 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431293 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431345 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431398 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.431436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.431422 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727zg\" (UniqueName: \"kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532638 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532673 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-727zg\" (UniqueName: \"kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532705 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532734 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532872 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532927 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.532979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.532970 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.533513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.533481 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.533643 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.533574 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.533643 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.533589 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.533750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.533691 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.535303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.535273 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.535397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.535379 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.541445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.541420 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-727zg\" (UniqueName: \"kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg\") pod \"console-7d7c8859fd-m2d2j\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.675447 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.675414 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:15.947757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:15.947675 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:20:15.951093 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:20:15.951065 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1833dc_cd66_4adb_94c9_e7af666b2f8e.slice/crio-c12744f7011e68faa5bdee39d9f54923c59f421d35ddc948a672ad70a3ab317c WatchSource:0}: Error finding container c12744f7011e68faa5bdee39d9f54923c59f421d35ddc948a672ad70a3ab317c: Status 404 returned error can't find the container with id c12744f7011e68faa5bdee39d9f54923c59f421d35ddc948a672ad70a3ab317c Sep 15 21:20:16.846562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:16.846528 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c8859fd-m2d2j" event={"ID":"fd1833dc-cd66-4adb-94c9-e7af666b2f8e","Type":"ContainerStarted","Data":"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f"} Sep 15 21:20:16.846962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:16.846579 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c8859fd-m2d2j" event={"ID":"fd1833dc-cd66-4adb-94c9-e7af666b2f8e","Type":"ContainerStarted","Data":"c12744f7011e68faa5bdee39d9f54923c59f421d35ddc948a672ad70a3ab317c"} Sep 15 21:20:16.864610 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:16.864566 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d7c8859fd-m2d2j" podStartSLOduration=1.864549689 podStartE2EDuration="1.864549689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:16.863505467 +0000 UTC m=+257.498657577" watchObservedRunningTime="2026-09-15 21:20:16.864549689 +0000 UTC m=+257.499701803" Sep 15 21:20:25.676611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:25.676514 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:25.676611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:25.676600 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:25.681406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:25.681381 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:25.881414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:25.881386 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:20:25.926509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:25.926476 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:20:50.950930 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:50.950872 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b44dc475c-mj4d5" podUID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" containerName="console" containerID="cri-o://e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9" gracePeriod=15 Sep 15 21:20:51.188821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.188797 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:20:51.306124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306041 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306097 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306130 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306154 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306172 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306198 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306344 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306229 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9459\" (UniqueName: \"kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459\") pod \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\" (UID: \"1e2d4acc-facd-4aa2-8b92-94c778e0139e\") " Sep 15 21:20:51.306816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306598 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:20:51.306816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306661 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:20:51.306816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306676 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:20:51.306816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.306700 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config" (OuterVolumeSpecName: "console-config") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:20:51.308387 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.308362 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459" (OuterVolumeSpecName: "kube-api-access-r9459") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "kube-api-access-r9459". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:20:51.308639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.308617 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:20:51.308639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.308627 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e2d4acc-facd-4aa2-8b92-94c778e0139e" (UID: "1e2d4acc-facd-4aa2-8b92-94c778e0139e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:20:51.407822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407779 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-oauth-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.407822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407816 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.407822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407853 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-trusted-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.408083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407867 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.408083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407880 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d4acc-facd-4aa2-8b92-94c778e0139e-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.408083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407892 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9459\" (UniqueName: \"kubernetes.io/projected/1e2d4acc-facd-4aa2-8b92-94c778e0139e-kube-api-access-r9459\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.408083 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.407903 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e2d4acc-facd-4aa2-8b92-94c778e0139e-console-oauth-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:20:51.973649 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.973614 2552 generic.go:356] "Generic (PLEG): container finished" podID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" containerID="e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9" exitCode=0 Sep 15 21:20:51.974157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.973661 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b44dc475c-mj4d5" Sep 15 21:20:51.974157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.973690 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b44dc475c-mj4d5" event={"ID":"1e2d4acc-facd-4aa2-8b92-94c778e0139e","Type":"ContainerDied","Data":"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9"} Sep 15 21:20:51.974157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.973751 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b44dc475c-mj4d5" event={"ID":"1e2d4acc-facd-4aa2-8b92-94c778e0139e","Type":"ContainerDied","Data":"9c40f7454aa36ff4bda15c4e0da281970cc89d8115213e1c9f29111f4668d2f8"} Sep 15 21:20:51.974157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.973773 2552 scope.go:117] "RemoveContainer" containerID="e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9" Sep 15 21:20:51.981439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.981422 2552 scope.go:117] "RemoveContainer" containerID="e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9" Sep 15 21:20:51.981708 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:20:51.981689 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9\": container with ID starting with e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9 not found: ID does not exist" containerID="e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9" Sep 15 21:20:51.981786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.981721 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9"} err="failed to get container status \"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9\": rpc error: code = NotFound desc = could not find container \"e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9\": container with ID starting with e6cb2da447a22a72154964fe8d65a28ac3bb9863745d26ea738e5b4bb48289d9 not found: ID does not exist" Sep 15 21:20:51.988448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.988423 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:20:51.990172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:51.990153 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b44dc475c-mj4d5"] Sep 15 21:20:53.789189 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:53.789158 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" path="/var/lib/kubelet/pods/1e2d4acc-facd-4aa2-8b92-94c778e0139e/volumes" Sep 15 21:20:59.703203 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:59.703173 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:20:59.705629 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:59.705605 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:20:59.713461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:20:59.713442 2552 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 21:21:17.468124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.468087 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd"] Sep 15 21:21:17.469503 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.468380 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" containerName="console" Sep 15 21:21:17.469503 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.468393 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" containerName="console" Sep 15 21:21:17.469503 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.468453 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2d4acc-facd-4aa2-8b92-94c778e0139e" containerName="console" Sep 15 21:21:17.470313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.470295 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.472289 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.472268 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 21:21:17.472424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.472409 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zznjp\"" Sep 15 21:21:17.472483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.472444 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 21:21:17.479368 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.479343 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd"] Sep 15 21:21:17.491191 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.491166 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjc8s\" (UniqueName: \"kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.491289 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.491212 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.491289 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.491262 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.592022 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.591978 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.592278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.592032 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.592278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.592104 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjc8s\" (UniqueName: \"kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.592405 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.592331 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.592405 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.592367 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.601023 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.600990 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjc8s\" (UniqueName: \"kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:17.779854 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:17.779749 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:18.049723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:18.049639 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd"] Sep 15 21:21:18.053035 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:21:18.053007 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2a3d2a_1d35_4c76_b063_536c86d62f27.slice/crio-d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42 WatchSource:0}: Error finding container d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42: Status 404 returned error can't find the container with id d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42 Sep 15 21:21:18.055386 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:18.055369 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:21:18.065499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:18.065472 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" event={"ID":"6d2a3d2a-1d35-4c76-b063-536c86d62f27","Type":"ContainerStarted","Data":"d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42"} Sep 15 21:21:26.098480 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:26.098439 2552 generic.go:356] "Generic (PLEG): container finished" podID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerID="8e5872e44aeca49633798aa1e258cc54a6e3dc0ac65ffd970c463ea87199ebbc" exitCode=0 Sep 15 21:21:26.098886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:26.098527 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" event={"ID":"6d2a3d2a-1d35-4c76-b063-536c86d62f27","Type":"ContainerDied","Data":"8e5872e44aeca49633798aa1e258cc54a6e3dc0ac65ffd970c463ea87199ebbc"} Sep 15 21:21:29.111687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:29.111648 2552 generic.go:356] "Generic (PLEG): container finished" podID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerID="79d64e3be9470abcf1de73093c81bcf8ec9c77f9453602fdc37e16aab8189371" exitCode=0 Sep 15 21:21:29.112078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:29.111710 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" event={"ID":"6d2a3d2a-1d35-4c76-b063-536c86d62f27","Type":"ContainerDied","Data":"79d64e3be9470abcf1de73093c81bcf8ec9c77f9453602fdc37e16aab8189371"} Sep 15 21:21:39.151676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:39.151643 2552 generic.go:356] "Generic (PLEG): container finished" podID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerID="bf780cf23794baa0d665819384dc1df8b542f2fe96385bb50a6f92306b75852e" exitCode=0 Sep 15 21:21:39.152061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:39.151738 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" event={"ID":"6d2a3d2a-1d35-4c76-b063-536c86d62f27","Type":"ContainerDied","Data":"bf780cf23794baa0d665819384dc1df8b542f2fe96385bb50a6f92306b75852e"} Sep 15 21:21:40.295437 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.295407 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:40.374994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.374958 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util\") pod \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " Sep 15 21:21:40.375153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.375009 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle\") pod \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " Sep 15 21:21:40.375153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.375089 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjc8s\" (UniqueName: \"kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s\") pod \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\" (UID: \"6d2a3d2a-1d35-4c76-b063-536c86d62f27\") " Sep 15 21:21:40.375646 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.375623 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle" (OuterVolumeSpecName: "bundle") pod "6d2a3d2a-1d35-4c76-b063-536c86d62f27" (UID: "6d2a3d2a-1d35-4c76-b063-536c86d62f27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:40.377323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.377287 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s" (OuterVolumeSpecName: "kube-api-access-kjc8s") pod "6d2a3d2a-1d35-4c76-b063-536c86d62f27" (UID: "6d2a3d2a-1d35-4c76-b063-536c86d62f27"). InnerVolumeSpecName "kube-api-access-kjc8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:21:40.379952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.379932 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util" (OuterVolumeSpecName: "util") pod "6d2a3d2a-1d35-4c76-b063-536c86d62f27" (UID: "6d2a3d2a-1d35-4c76-b063-536c86d62f27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:40.475881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.475781 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjc8s\" (UniqueName: \"kubernetes.io/projected/6d2a3d2a-1d35-4c76-b063-536c86d62f27-kube-api-access-kjc8s\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:40.475881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.475814 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:40.475881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:40.475823 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d2a3d2a-1d35-4c76-b063-536c86d62f27-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:41.161908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:41.161880 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" Sep 15 21:21:41.161908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:41.161893 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p76gd" event={"ID":"6d2a3d2a-1d35-4c76-b063-536c86d62f27","Type":"ContainerDied","Data":"d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42"} Sep 15 21:21:41.162104 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:41.161946 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43ec513446b23471543694427c6395e95819bf556b1c8c62d29300977af2a42" Sep 15 21:21:45.584627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584592 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6"] Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584870 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="pull" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584883 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="pull" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584897 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="util" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584904 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="util" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584911 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="extract" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584916 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="extract" Sep 15 21:21:45.585005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.584964 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2a3d2a-1d35-4c76-b063-536c86d62f27" containerName="extract" Sep 15 21:21:45.586884 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.586869 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.589044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.589020 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:21:45.589162 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.589119 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jdkhs\"" Sep 15 21:21:45.589501 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.589485 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 15 21:21:45.603866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.603816 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6"] Sep 15 21:21:45.718289 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.718255 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc29ec1b-3633-4642-b0d5-698078feac83-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.718432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.718302 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6w8\" (UniqueName: \"kubernetes.io/projected/bc29ec1b-3633-4642-b0d5-698078feac83-kube-api-access-dm6w8\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.819661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.819629 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc29ec1b-3633-4642-b0d5-698078feac83-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.819803 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.819676 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6w8\" (UniqueName: \"kubernetes.io/projected/bc29ec1b-3633-4642-b0d5-698078feac83-kube-api-access-dm6w8\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.820008 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.819989 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc29ec1b-3633-4642-b0d5-698078feac83-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.833653 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.833631 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6w8\" (UniqueName: \"kubernetes.io/projected/bc29ec1b-3633-4642-b0d5-698078feac83-kube-api-access-dm6w8\") pod \"cert-manager-operator-controller-manager-55c444bf86-hxbj6\" (UID: \"bc29ec1b-3633-4642-b0d5-698078feac83\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:45.895290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:45.895261 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" Sep 15 21:21:46.182615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:46.182580 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6"] Sep 15 21:21:46.184881 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:21:46.184858 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc29ec1b_3633_4642_b0d5_698078feac83.slice/crio-b18360f1040bdcf922d30ceada53ff0bdbbb4d77396a6906ddea104baf79b808 WatchSource:0}: Error finding container b18360f1040bdcf922d30ceada53ff0bdbbb4d77396a6906ddea104baf79b808: Status 404 returned error can't find the container with id b18360f1040bdcf922d30ceada53ff0bdbbb4d77396a6906ddea104baf79b808 Sep 15 21:21:47.186343 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:47.186302 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" event={"ID":"bc29ec1b-3633-4642-b0d5-698078feac83","Type":"ContainerStarted","Data":"b18360f1040bdcf922d30ceada53ff0bdbbb4d77396a6906ddea104baf79b808"} Sep 15 21:21:48.192426 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:48.192386 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" event={"ID":"bc29ec1b-3633-4642-b0d5-698078feac83","Type":"ContainerStarted","Data":"3f922c3a2db2329dcae7e8879cec10ec92cf376fae3243276eb8ccab87387693"} Sep 15 21:21:48.222427 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:48.222372 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hxbj6" podStartSLOduration=1.322720705 podStartE2EDuration="3.222356065s" totalImagesPullingTime="1.89963536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:48.220412285 +0000 UTC m=+348.855564397" watchObservedRunningTime="2026-09-15 21:21:48.222356065 +0000 UTC m=+348.857508176" Sep 15 21:21:49.268644 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.268608 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4"] Sep 15 21:21:49.271036 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.271020 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.273638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.273611 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 21:21:49.273777 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.273614 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zznjp\"" Sep 15 21:21:49.274053 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.274037 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 21:21:49.283856 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.283814 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4"] Sep 15 21:21:49.349853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.349802 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.350040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.349862 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pvh\" (UniqueName: \"kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.350040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.349956 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.450667 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.450630 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.450861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.450673 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pvh\" (UniqueName: \"kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.450861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.450793 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.451133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.451112 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.451205 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.451129 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.460279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.460249 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pvh\" (UniqueName: \"kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.579926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.579823 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:49.855015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:49.854937 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4"] Sep 15 21:21:49.857718 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:21:49.857690 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384ac9f0_f7ed_4f0d_8980_adf83b60a265.slice/crio-232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585 WatchSource:0}: Error finding container 232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585: Status 404 returned error can't find the container with id 232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585 Sep 15 21:21:50.201382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:50.201350 2552 generic.go:356] "Generic (PLEG): container finished" podID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerID="43c1e78d6ef1b158603161d34dfb5826865160da7027357111951217a9a31651" exitCode=0 Sep 15 21:21:50.201561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:50.201429 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" event={"ID":"384ac9f0-f7ed-4f0d-8980-adf83b60a265","Type":"ContainerDied","Data":"43c1e78d6ef1b158603161d34dfb5826865160da7027357111951217a9a31651"} Sep 15 21:21:50.201561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:50.201492 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" event={"ID":"384ac9f0-f7ed-4f0d-8980-adf83b60a265","Type":"ContainerStarted","Data":"232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585"} Sep 15 21:21:51.770107 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.770071 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96txg"] Sep 15 21:21:51.772633 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.772592 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.776350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.776165 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 21:21:51.776464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.776355 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 21:21:51.797429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.797399 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96txg"] Sep 15 21:21:51.872172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.872132 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25ln\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-kube-api-access-z25ln\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.872172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.872176 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.973379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.973337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25ln\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-kube-api-access-z25ln\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.973379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.973377 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.986641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.986606 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:51.986862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:51.986817 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25ln\" (UniqueName: \"kubernetes.io/projected/29c6b5a5-4460-479f-b43b-743f6ec12608-kube-api-access-z25ln\") pod \"cert-manager-cainjector-559cdd8c87-96txg\" (UID: \"29c6b5a5-4460-479f-b43b-743f6ec12608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:52.084386 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:52.084283 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" Sep 15 21:21:52.765555 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:52.765532 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96txg"] Sep 15 21:21:52.768022 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:21:52.767995 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c6b5a5_4460_479f_b43b_743f6ec12608.slice/crio-ca02420bd1e936199678ff0902b202cb7893ea3210368d85ab9b87f51b74f8a6 WatchSource:0}: Error finding container ca02420bd1e936199678ff0902b202cb7893ea3210368d85ab9b87f51b74f8a6: Status 404 returned error can't find the container with id ca02420bd1e936199678ff0902b202cb7893ea3210368d85ab9b87f51b74f8a6 Sep 15 21:21:53.215278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:53.215246 2552 generic.go:356] "Generic (PLEG): container finished" podID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerID="0bda1ed8f9222f7969778f32cc3d725129ab04a7a423121f4f9b893d72d40a8c" exitCode=0 Sep 15 21:21:53.215713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:53.215334 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" event={"ID":"384ac9f0-f7ed-4f0d-8980-adf83b60a265","Type":"ContainerDied","Data":"0bda1ed8f9222f7969778f32cc3d725129ab04a7a423121f4f9b893d72d40a8c"} Sep 15 21:21:53.216630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:53.216605 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" event={"ID":"29c6b5a5-4460-479f-b43b-743f6ec12608","Type":"ContainerStarted","Data":"ca02420bd1e936199678ff0902b202cb7893ea3210368d85ab9b87f51b74f8a6"} Sep 15 21:21:54.224155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:54.224121 2552 generic.go:356] "Generic (PLEG): container finished" podID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerID="42f2bc7ea3d6c6dfa9525a74bbc8031c2e15393affa8d4a5e81a7b5f687986eb" exitCode=0 Sep 15 21:21:54.224562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:54.224206 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" event={"ID":"384ac9f0-f7ed-4f0d-8980-adf83b60a265","Type":"ContainerDied","Data":"42f2bc7ea3d6c6dfa9525a74bbc8031c2e15393affa8d4a5e81a7b5f687986eb"} Sep 15 21:21:55.551652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.551627 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:55.705879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.705844 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76pvh\" (UniqueName: \"kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh\") pod \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " Sep 15 21:21:55.706066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.705932 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle\") pod \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " Sep 15 21:21:55.706066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.705992 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util\") pod \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\" (UID: \"384ac9f0-f7ed-4f0d-8980-adf83b60a265\") " Sep 15 21:21:55.706294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.706258 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle" (OuterVolumeSpecName: "bundle") pod "384ac9f0-f7ed-4f0d-8980-adf83b60a265" (UID: "384ac9f0-f7ed-4f0d-8980-adf83b60a265"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:55.708025 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.707986 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh" (OuterVolumeSpecName: "kube-api-access-76pvh") pod "384ac9f0-f7ed-4f0d-8980-adf83b60a265" (UID: "384ac9f0-f7ed-4f0d-8980-adf83b60a265"). InnerVolumeSpecName "kube-api-access-76pvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:21:55.710547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.710524 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util" (OuterVolumeSpecName: "util") pod "384ac9f0-f7ed-4f0d-8980-adf83b60a265" (UID: "384ac9f0-f7ed-4f0d-8980-adf83b60a265"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:21:55.807317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.807291 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:55.807317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.807315 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384ac9f0-f7ed-4f0d-8980-adf83b60a265-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:55.807518 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:55.807324 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76pvh\" (UniqueName: \"kubernetes.io/projected/384ac9f0-f7ed-4f0d-8980-adf83b60a265-kube-api-access-76pvh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:21:56.234232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:56.234192 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" event={"ID":"29c6b5a5-4460-479f-b43b-743f6ec12608","Type":"ContainerStarted","Data":"f525e7b07f7973cf173dd754bb647284a0aec5195d2ad403fb5cc5a353936fea"} Sep 15 21:21:56.236088 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:56.236060 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" event={"ID":"384ac9f0-f7ed-4f0d-8980-adf83b60a265","Type":"ContainerDied","Data":"232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585"} Sep 15 21:21:56.236203 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:56.236101 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232dda9f43917ce5763aed2bab3864c573ed686ec7c5f9f0c60d89e8326f2585" Sep 15 21:21:56.236203 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:56.236075 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlp4" Sep 15 21:21:56.249436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:21:56.249390 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96txg" podStartSLOduration=2.446158698 podStartE2EDuration="5.249375714s" totalImagesPullingTime="2.803217016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:56.248210334 +0000 UTC m=+356.883362456" watchObservedRunningTime="2026-09-15 21:21:56.249375714 +0000 UTC m=+356.884527824" Sep 15 21:22:03.546572 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546534 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd"] Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546855 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="extract" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546870 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="extract" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546883 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="pull" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546889 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="pull" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546896 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="util" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546902 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="util" Sep 15 21:22:03.546974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.546957 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="384ac9f0-f7ed-4f0d-8980-adf83b60a265" containerName="extract" Sep 15 21:22:03.584307 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.584278 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd"] Sep 15 21:22:03.584459 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.584369 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.587938 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.587915 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wjd7b\"" Sep 15 21:22:03.588500 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.588483 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:22:03.588569 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.588504 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 21:22:03.670611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.670572 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6726\" (UniqueName: \"kubernetes.io/projected/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-kube-api-access-d6726\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.670769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.670628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.771115 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.771075 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6726\" (UniqueName: \"kubernetes.io/projected/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-kube-api-access-d6726\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.771307 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.771124 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.771487 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.771468 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.779551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.779530 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6726\" (UniqueName: \"kubernetes.io/projected/fb4888a5-1b7d-42e5-b85f-9a8a257007b2-kube-api-access-d6726\") pod \"openshift-lws-operator-fd8ccff4c-fr2sd\" (UID: \"fb4888a5-1b7d-42e5-b85f-9a8a257007b2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:03.893587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:03.893557 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" Sep 15 21:22:04.181182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:04.181154 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd"] Sep 15 21:22:04.183731 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:04.183703 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4888a5_1b7d_42e5_b85f_9a8a257007b2.slice/crio-d68b7f7a6fa26388066593e79c747add19c90675fa8fa420b279c9c8129ad530 WatchSource:0}: Error finding container d68b7f7a6fa26388066593e79c747add19c90675fa8fa420b279c9c8129ad530: Status 404 returned error can't find the container with id d68b7f7a6fa26388066593e79c747add19c90675fa8fa420b279c9c8129ad530 Sep 15 21:22:04.269597 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:04.269563 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" event={"ID":"fb4888a5-1b7d-42e5-b85f-9a8a257007b2","Type":"ContainerStarted","Data":"d68b7f7a6fa26388066593e79c747add19c90675fa8fa420b279c9c8129ad530"} Sep 15 21:22:07.285499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:07.285458 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" event={"ID":"fb4888a5-1b7d-42e5-b85f-9a8a257007b2","Type":"ContainerStarted","Data":"462b1453b76c1d0f01574300b77d73fb1844c807e3b8a7ab66ad95ee84192d01"} Sep 15 21:22:07.302935 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:07.302880 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fr2sd" podStartSLOduration=1.991851956 podStartE2EDuration="4.302862435s" totalImagesPullingTime="2.311010479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:07.300986128 +0000 UTC m=+367.936138241" watchObservedRunningTime="2026-09-15 21:22:07.302862435 +0000 UTC m=+367.938014545" Sep 15 21:22:09.612240 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.612208 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx"] Sep 15 21:22:09.614596 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.614580 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.616968 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.616943 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 21:22:09.617409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.617387 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 21:22:09.617520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.617387 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zznjp\"" Sep 15 21:22:09.623700 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.623680 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx"] Sep 15 21:22:09.723658 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.723624 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.723816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.723677 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfbz\" (UniqueName: \"kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.723816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.723750 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.824476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.824443 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.824476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.824481 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfbz\" (UniqueName: \"kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.824743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.824502 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.824865 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.824817 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.824951 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.824931 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.833273 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.833244 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfbz\" (UniqueName: \"kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:09.924532 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:09.924500 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:10.194183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:10.194155 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx"] Sep 15 21:22:10.196300 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:10.196268 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5187dd_3468_411f_af4c_bae12ae760ae.slice/crio-db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687 WatchSource:0}: Error finding container db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687: Status 404 returned error can't find the container with id db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687 Sep 15 21:22:10.300469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:10.300438 2552 generic.go:356] "Generic (PLEG): container finished" podID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerID="c9222857ef0b0540823b55829bbc65fb1c700c93847117bcb46ece073ea2a080" exitCode=0 Sep 15 21:22:10.300582 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:10.300481 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" event={"ID":"0f5187dd-3468-411f-af4c-bae12ae760ae","Type":"ContainerDied","Data":"c9222857ef0b0540823b55829bbc65fb1c700c93847117bcb46ece073ea2a080"} Sep 15 21:22:10.300582 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:10.300520 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" event={"ID":"0f5187dd-3468-411f-af4c-bae12ae760ae","Type":"ContainerStarted","Data":"db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687"} Sep 15 21:22:11.309994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:11.309958 2552 generic.go:356] "Generic (PLEG): container finished" podID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerID="93edd871cfea18b51fcd2360919d4895bdfa22cbbc5cc46a74523314056c700f" exitCode=0 Sep 15 21:22:11.310432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:11.310008 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" event={"ID":"0f5187dd-3468-411f-af4c-bae12ae760ae","Type":"ContainerDied","Data":"93edd871cfea18b51fcd2360919d4895bdfa22cbbc5cc46a74523314056c700f"} Sep 15 21:22:12.316742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:12.316706 2552 generic.go:356] "Generic (PLEG): container finished" podID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerID="5d0345d53b9062177d2d564bdc93e94c23a20425b009f57ca37125e6eb0533c7" exitCode=0 Sep 15 21:22:12.317164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:12.316765 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" event={"ID":"0f5187dd-3468-411f-af4c-bae12ae760ae","Type":"ContainerDied","Data":"5d0345d53b9062177d2d564bdc93e94c23a20425b009f57ca37125e6eb0533c7"} Sep 15 21:22:13.455883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.455858 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:13.555384 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.555345 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swfbz\" (UniqueName: \"kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz\") pod \"0f5187dd-3468-411f-af4c-bae12ae760ae\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " Sep 15 21:22:13.555569 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.555429 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle\") pod \"0f5187dd-3468-411f-af4c-bae12ae760ae\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " Sep 15 21:22:13.555569 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.555464 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util\") pod \"0f5187dd-3468-411f-af4c-bae12ae760ae\" (UID: \"0f5187dd-3468-411f-af4c-bae12ae760ae\") " Sep 15 21:22:13.556123 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.556089 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle" (OuterVolumeSpecName: "bundle") pod "0f5187dd-3468-411f-af4c-bae12ae760ae" (UID: "0f5187dd-3468-411f-af4c-bae12ae760ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:22:13.557414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.557394 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz" (OuterVolumeSpecName: "kube-api-access-swfbz") pod "0f5187dd-3468-411f-af4c-bae12ae760ae" (UID: "0f5187dd-3468-411f-af4c-bae12ae760ae"). InnerVolumeSpecName "kube-api-access-swfbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:22:13.561109 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.561079 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util" (OuterVolumeSpecName: "util") pod "0f5187dd-3468-411f-af4c-bae12ae760ae" (UID: "0f5187dd-3468-411f-af4c-bae12ae760ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:22:13.656147 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.656105 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:13.656147 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.656143 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f5187dd-3468-411f-af4c-bae12ae760ae-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:13.656350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:13.656156 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfbz\" (UniqueName: \"kubernetes.io/projected/0f5187dd-3468-411f-af4c-bae12ae760ae-kube-api-access-swfbz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:14.327850 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:14.327804 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" Sep 15 21:22:14.328041 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:14.327804 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwmddx" event={"ID":"0f5187dd-3468-411f-af4c-bae12ae760ae","Type":"ContainerDied","Data":"db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687"} Sep 15 21:22:14.328041 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:14.327942 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db74eed5e772900d8dffb7d61f14b0e0abe46610209b3ae1c1cdc0a39ec11687" Sep 15 21:22:21.852509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852470 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb"] Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852744 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="util" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852755 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="util" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852764 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="pull" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852769 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="pull" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852780 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="extract" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852787 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="extract" Sep 15 21:22:21.853043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.852864 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5187dd-3468-411f-af4c-bae12ae760ae" containerName="extract" Sep 15 21:22:21.855499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.855478 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:21.857655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.857631 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 21:22:21.858002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.857985 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zznjp\"" Sep 15 21:22:21.858340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.858323 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 21:22:21.876667 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.876644 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb"] Sep 15 21:22:21.919440 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.919407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:21.919605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.919464 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:21.919605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:21.919493 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47jp\" (UniqueName: \"kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.020248 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.020217 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.020415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.020272 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.020415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.020292 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47jp\" (UniqueName: \"kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.020638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.020617 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.020674 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.020623 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.028993 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.028949 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47jp\" (UniqueName: \"kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.164813 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.164775 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:22.444659 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:22.444634 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb"] Sep 15 21:22:22.447184 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:22.447149 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2827168_e008_4a5a_a335_8617b59e7905.slice/crio-9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2 WatchSource:0}: Error finding container 9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2: Status 404 returned error can't find the container with id 9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2 Sep 15 21:22:23.334776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.334694 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm"] Sep 15 21:22:23.338945 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.338919 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.344634 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.344607 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 21:22:23.344764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.344698 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 21:22:23.344764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.344742 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 21:22:23.346784 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.346764 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 21:22:23.349358 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.349340 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p85r2\"" Sep 15 21:22:23.365324 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.365291 2552 generic.go:356] "Generic (PLEG): container finished" podID="f2827168-e008-4a5a-a335-8617b59e7905" containerID="22048bc5e8b7fe2cffa5bbf73e6be8b68ec801a133a5812161f9e286ea5eaf23" exitCode=0 Sep 15 21:22:23.365463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.365390 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" event={"ID":"f2827168-e008-4a5a-a335-8617b59e7905","Type":"ContainerDied","Data":"22048bc5e8b7fe2cffa5bbf73e6be8b68ec801a133a5812161f9e286ea5eaf23"} Sep 15 21:22:23.365463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.365444 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" event={"ID":"f2827168-e008-4a5a-a335-8617b59e7905","Type":"ContainerStarted","Data":"9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2"} Sep 15 21:22:23.369366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.369342 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm"] Sep 15 21:22:23.431058 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.431024 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.431058 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.431057 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.431262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.431134 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nl4z\" (UniqueName: \"kubernetes.io/projected/1f75cdf0-2576-47b7-a6bc-978931526051-kube-api-access-6nl4z\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.431262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.431162 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.431262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.431191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.531517 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.531482 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.531517 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.531520 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.531724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.531563 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nl4z\" (UniqueName: \"kubernetes.io/projected/1f75cdf0-2576-47b7-a6bc-978931526051-kube-api-access-6nl4z\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.531724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.531584 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.531724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.531608 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.532152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.532132 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.533954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.533925 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f75cdf0-2576-47b7-a6bc-978931526051-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.534107 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.534088 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.534222 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.534203 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f75cdf0-2576-47b7-a6bc-978931526051-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.542860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.542820 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nl4z\" (UniqueName: \"kubernetes.io/projected/1f75cdf0-2576-47b7-a6bc-978931526051-kube-api-access-6nl4z\") pod \"opendatahub-operator-controller-manager-596bc79d9b-ltcrm\" (UID: \"1f75cdf0-2576-47b7-a6bc-978931526051\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.649089 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.649054 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:23.940798 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:23.940774 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm"] Sep 15 21:22:23.942304 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:23.942276 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f75cdf0_2576_47b7_a6bc_978931526051.slice/crio-b6f6c0f89fa15e4d9beeab400bc6ac85be655b978687357367c923eb16d96eae WatchSource:0}: Error finding container b6f6c0f89fa15e4d9beeab400bc6ac85be655b978687357367c923eb16d96eae: Status 404 returned error can't find the container with id b6f6c0f89fa15e4d9beeab400bc6ac85be655b978687357367c923eb16d96eae Sep 15 21:22:24.372518 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:24.372481 2552 generic.go:356] "Generic (PLEG): container finished" podID="f2827168-e008-4a5a-a335-8617b59e7905" containerID="c67e9ffffcbae8e3cb4b97d53e8c84a05122795bd4e83c452d9d4e465a319da3" exitCode=0 Sep 15 21:22:24.372986 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:24.372569 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" event={"ID":"f2827168-e008-4a5a-a335-8617b59e7905","Type":"ContainerDied","Data":"c67e9ffffcbae8e3cb4b97d53e8c84a05122795bd4e83c452d9d4e465a319da3"} Sep 15 21:22:24.374345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:24.374218 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" event={"ID":"1f75cdf0-2576-47b7-a6bc-978931526051","Type":"ContainerStarted","Data":"b6f6c0f89fa15e4d9beeab400bc6ac85be655b978687357367c923eb16d96eae"} Sep 15 21:22:25.382456 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:25.382417 2552 generic.go:356] "Generic (PLEG): container finished" podID="f2827168-e008-4a5a-a335-8617b59e7905" containerID="723751a45c816da1fefd2d00a5e4be81df2fdc4d37533ddb372f6d0cc3286ab0" exitCode=0 Sep 15 21:22:25.382876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:25.382495 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" event={"ID":"f2827168-e008-4a5a-a335-8617b59e7905","Type":"ContainerDied","Data":"723751a45c816da1fefd2d00a5e4be81df2fdc4d37533ddb372f6d0cc3286ab0"} Sep 15 21:22:26.854727 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.854703 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:26.958416 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.958390 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util\") pod \"f2827168-e008-4a5a-a335-8617b59e7905\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " Sep 15 21:22:26.958554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.958443 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n47jp\" (UniqueName: \"kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp\") pod \"f2827168-e008-4a5a-a335-8617b59e7905\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " Sep 15 21:22:26.958554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.958473 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle\") pod \"f2827168-e008-4a5a-a335-8617b59e7905\" (UID: \"f2827168-e008-4a5a-a335-8617b59e7905\") " Sep 15 21:22:26.959250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.959209 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle" (OuterVolumeSpecName: "bundle") pod "f2827168-e008-4a5a-a335-8617b59e7905" (UID: "f2827168-e008-4a5a-a335-8617b59e7905"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:22:26.960375 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.960353 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp" (OuterVolumeSpecName: "kube-api-access-n47jp") pod "f2827168-e008-4a5a-a335-8617b59e7905" (UID: "f2827168-e008-4a5a-a335-8617b59e7905"). InnerVolumeSpecName "kube-api-access-n47jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:22:26.963223 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:26.963204 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util" (OuterVolumeSpecName: "util") pod "f2827168-e008-4a5a-a335-8617b59e7905" (UID: "f2827168-e008-4a5a-a335-8617b59e7905"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:22:27.059152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.059118 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:27.059152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.059149 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47jp\" (UniqueName: \"kubernetes.io/projected/f2827168-e008-4a5a-a335-8617b59e7905-kube-api-access-n47jp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:27.059152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.059160 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2827168-e008-4a5a-a335-8617b59e7905-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:22:27.394212 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.394181 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" Sep 15 21:22:27.394473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.394182 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zd5bb" event={"ID":"f2827168-e008-4a5a-a335-8617b59e7905","Type":"ContainerDied","Data":"9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2"} Sep 15 21:22:27.394473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.394317 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e77b82247535979c6c18596fec2805f2834524512d229a634051c8f9cb977a2" Sep 15 21:22:27.396033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:27.396008 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" event={"ID":"1f75cdf0-2576-47b7-a6bc-978931526051","Type":"ContainerStarted","Data":"a99c1d1e6d2419f6e2c5315b102f5fb2e22ba08edbd0a6de5cb0ddb5e07bd0ff"} Sep 15 21:22:28.402347 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:28.402309 2552 generic.go:356] "Generic (PLEG): container finished" podID="1f75cdf0-2576-47b7-a6bc-978931526051" containerID="a99c1d1e6d2419f6e2c5315b102f5fb2e22ba08edbd0a6de5cb0ddb5e07bd0ff" exitCode=0 Sep 15 21:22:28.402705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:28.402403 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" event={"ID":"1f75cdf0-2576-47b7-a6bc-978931526051","Type":"ContainerDied","Data":"a99c1d1e6d2419f6e2c5315b102f5fb2e22ba08edbd0a6de5cb0ddb5e07bd0ff"} Sep 15 21:22:29.409354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:29.409311 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" event={"ID":"1f75cdf0-2576-47b7-a6bc-978931526051","Type":"ContainerStarted","Data":"d6e028d95c4e9509d28144e498922c189e48b35e78cc1493e687883b3c5412a7"} Sep 15 21:22:29.409729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:29.409432 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:29.449987 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:29.449932 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" podStartSLOduration=2.066866268 podStartE2EDuration="6.44991172s" totalImagesPullingTime="3.368528431s" totalInitContainerRuntime="1.014517021s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:29.447626455 +0000 UTC m=+390.082778588" watchObservedRunningTime="2026-09-15 21:22:29.44991172 +0000 UTC m=+390.085063831" Sep 15 21:22:39.012079 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012046 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d947dbf67-qlzjl"] Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012353 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="pull" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012366 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="pull" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012376 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="util" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012381 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="util" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012387 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="extract" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012392 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="extract" Sep 15 21:22:39.012464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.012451 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2827168-e008-4a5a-a335-8617b59e7905" containerName="extract" Sep 15 21:22:39.116406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.116366 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d947dbf67-qlzjl"] Sep 15 21:22:39.116559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.116475 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158173 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158141 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-trusted-ca-bundle\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-console-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158220 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158261 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-oauth-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158275 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-service-ca\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158298 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-oauth-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.158495 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.158358 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4bg\" (UniqueName: \"kubernetes.io/projected/6daa122c-af09-4312-812e-44da8b4fd9e0-kube-api-access-hh4bg\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259258 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-oauth-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259293 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-service-ca\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259311 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-oauth-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259338 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4bg\" (UniqueName: \"kubernetes.io/projected/6daa122c-af09-4312-812e-44da8b4fd9e0-kube-api-access-hh4bg\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259399 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-trusted-ca-bundle\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-console-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.259504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.259478 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.260169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.260144 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-service-ca\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.260295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.260183 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-oauth-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.260295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.260265 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-console-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.260295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.260284 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daa122c-af09-4312-812e-44da8b4fd9e0-trusted-ca-bundle\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.261914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.261886 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-oauth-config\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.261996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.261932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6daa122c-af09-4312-812e-44da8b4fd9e0-console-serving-cert\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.267071 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.267020 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4bg\" (UniqueName: \"kubernetes.io/projected/6daa122c-af09-4312-812e-44da8b4fd9e0-kube-api-access-hh4bg\") pod \"console-6d947dbf67-qlzjl\" (UID: \"6daa122c-af09-4312-812e-44da8b4fd9e0\") " pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.426236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.426200 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:39.728273 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:39.728225 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d947dbf67-qlzjl"] Sep 15 21:22:39.730956 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:39.730926 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daa122c_af09_4312_812e_44da8b4fd9e0.slice/crio-405332fc6401df22ca73d34f15f3db6f2adfd2de54dbba0d91a989a221336b80 WatchSource:0}: Error finding container 405332fc6401df22ca73d34f15f3db6f2adfd2de54dbba0d91a989a221336b80: Status 404 returned error can't find the container with id 405332fc6401df22ca73d34f15f3db6f2adfd2de54dbba0d91a989a221336b80 Sep 15 21:22:40.416280 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:40.416250 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-ltcrm" Sep 15 21:22:40.460383 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:40.460342 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d947dbf67-qlzjl" event={"ID":"6daa122c-af09-4312-812e-44da8b4fd9e0","Type":"ContainerStarted","Data":"92743eb15b7f0a74ee0f26b0bd7f731f808c6c7a0f0689ef131b843c28e4bb7f"} Sep 15 21:22:40.460543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:40.460405 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d947dbf67-qlzjl" event={"ID":"6daa122c-af09-4312-812e-44da8b4fd9e0","Type":"ContainerStarted","Data":"405332fc6401df22ca73d34f15f3db6f2adfd2de54dbba0d91a989a221336b80"} Sep 15 21:22:40.480886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:40.480812 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d947dbf67-qlzjl" podStartSLOduration=2.480796684 podStartE2EDuration="2.480796684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:40.47856652 +0000 UTC m=+401.113718631" watchObservedRunningTime="2026-09-15 21:22:40.480796684 +0000 UTC m=+401.115948794" Sep 15 21:22:42.984869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:42.984814 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4"] Sep 15 21:22:42.987750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:42.987717 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:42.989868 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:42.989844 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 15 21:22:42.990010 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:42.989994 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 15 21:22:43.013600 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.013567 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4"] Sep 15 21:22:43.089713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.089675 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjdm\" (UniqueName: \"kubernetes.io/projected/476f847b-6bd2-4b39-82cf-9cba3b64c97e-kube-api-access-cvjdm\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.089929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.089742 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tmp\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.089929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.089843 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.191149 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.191110 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjdm\" (UniqueName: \"kubernetes.io/projected/476f847b-6bd2-4b39-82cf-9cba3b64c97e-kube-api-access-cvjdm\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.191287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.191165 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tmp\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.191287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.191195 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.193457 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.193431 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tmp\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.193782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.193754 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476f847b-6bd2-4b39-82cf-9cba3b64c97e-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.202491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.202467 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjdm\" (UniqueName: \"kubernetes.io/projected/476f847b-6bd2-4b39-82cf-9cba3b64c97e-kube-api-access-cvjdm\") pod \"kube-auth-proxy-58bcc99dc6-4mrh4\" (UID: \"476f847b-6bd2-4b39-82cf-9cba3b64c97e\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.297339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.297257 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" Sep 15 21:22:43.593097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:43.593067 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4"] Sep 15 21:22:43.594783 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:43.594752 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476f847b_6bd2_4b39_82cf_9cba3b64c97e.slice/crio-2d791ddab953b7664149d9b0cd2240bfdf2394c1a1dcfbeca326a8bc90947d34 WatchSource:0}: Error finding container 2d791ddab953b7664149d9b0cd2240bfdf2394c1a1dcfbeca326a8bc90947d34: Status 404 returned error can't find the container with id 2d791ddab953b7664149d9b0cd2240bfdf2394c1a1dcfbeca326a8bc90947d34 Sep 15 21:22:44.488110 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.488027 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" event={"ID":"476f847b-6bd2-4b39-82cf-9cba3b64c97e","Type":"ContainerStarted","Data":"2d791ddab953b7664149d9b0cd2240bfdf2394c1a1dcfbeca326a8bc90947d34"} Sep 15 21:22:44.728340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.728297 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:22:44.733298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.733252 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.737398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.737050 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 15 21:22:44.737398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.737096 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 15 21:22:44.737398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.737160 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 15 21:22:44.737398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.737096 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 21:22:44.737398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.737238 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4jkpg\"" Sep 15 21:22:44.751287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.751259 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:22:44.804338 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804307 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804361 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804485 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804509 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7f6\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.804687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.804555 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.905788 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905728 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905795 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7f6\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905856 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905900 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905954 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.905985 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.906015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.906536 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.906500 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.909529 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.909505 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.909665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.909640 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.909909 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.909848 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.910025 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.910003 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.915896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.915870 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:44.918042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:44.918017 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7f6\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6\") pod \"istiod-openshift-gateway-7b7995d5cd-9h76f\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:45.048225 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:45.048138 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:46.498869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:46.498809 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" event={"ID":"476f847b-6bd2-4b39-82cf-9cba3b64c97e","Type":"ContainerStarted","Data":"0c928402ea2fb8db9f603040744731d1f0bbce847145f82384854f92227912ba"} Sep 15 21:22:46.514951 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:46.514898 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-4mrh4" podStartSLOduration=1.6799931909999999 podStartE2EDuration="4.51488059s" totalImagesPullingTime="2.834887399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:46.513529259 +0000 UTC m=+407.148681369" watchObservedRunningTime="2026-09-15 21:22:46.51488059 +0000 UTC m=+407.150032701" Sep 15 21:22:46.656810 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:46.656785 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:22:46.659135 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:22:46.659103 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66c738c_01ab_4380_94a2_385ebc63633b.slice/crio-b41378512a7dda6ebe86d13d63e5649cdfb64686d2803f16e7c93fafde88f314 WatchSource:0}: Error finding container b41378512a7dda6ebe86d13d63e5649cdfb64686d2803f16e7c93fafde88f314: Status 404 returned error can't find the container with id b41378512a7dda6ebe86d13d63e5649cdfb64686d2803f16e7c93fafde88f314 Sep 15 21:22:47.506371 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:47.506323 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" event={"ID":"c66c738c-01ab-4380-94a2-385ebc63633b","Type":"ContainerStarted","Data":"b41378512a7dda6ebe86d13d63e5649cdfb64686d2803f16e7c93fafde88f314"} Sep 15 21:22:49.130466 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.130426 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:22:49.130792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.130495 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:22:49.426883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.426776 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:49.426883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.426884 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:49.431872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.431847 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:49.517880 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.517848 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" event={"ID":"c66c738c-01ab-4380-94a2-385ebc63633b","Type":"ContainerStarted","Data":"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71"} Sep 15 21:22:49.518085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.517966 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:22:49.521765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.521742 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d947dbf67-qlzjl" Sep 15 21:22:49.538702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.538643 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" podStartSLOduration=3.069741138 podStartE2EDuration="5.538628161s" totalImagesPullingTime="2.468887023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:49.537990217 +0000 UTC m=+410.173142328" watchObservedRunningTime="2026-09-15 21:22:49.538628161 +0000 UTC m=+410.173780273" Sep 15 21:22:49.590208 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:49.590173 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:22:50.523723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:50.523692 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9h76f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:22:50.524075 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:50.523742 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:22:53.522992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:22:53.522954 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:23:00.485860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.485806 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc"] Sep 15 21:23:00.490589 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.490571 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.526460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.526426 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc"] Sep 15 21:23:00.537562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537571 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537593 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537647 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537693 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhv7q\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-kube-api-access-hhv7q\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537721 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53015761-220d-4d64-8b2f-d317680d9ffa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.537962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.537749 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639008 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639169 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639218 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639240 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639258 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639289 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhv7q\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-kube-api-access-hhv7q\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.639351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639324 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53015761-220d-4d64-8b2f-d317680d9ffa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.640010 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.639982 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.641556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.641529 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53015761-220d-4d64-8b2f-d317680d9ffa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.641556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.641541 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.641556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.641541 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.641759 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.641605 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53015761-220d-4d64-8b2f-d317680d9ffa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.648964 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.648932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.649589 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.649569 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhv7q\" (UniqueName: \"kubernetes.io/projected/53015761-220d-4d64-8b2f-d317680d9ffa-kube-api-access-hhv7q\") pod \"istiod-openshift-gateway-5677ccb49b-5s4kc\" (UID: \"53015761-220d-4d64-8b2f-d317680d9ffa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:00.800220 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:00.800129 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:01.093604 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.093578 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc"] Sep 15 21:23:01.095453 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:23:01.095428 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53015761_220d_4d64_8b2f_d317680d9ffa.slice/crio-59d438d277816140b3b45ad12600d8901889c71785dcdf0843f554e7efc41456 WatchSource:0}: Error finding container 59d438d277816140b3b45ad12600d8901889c71785dcdf0843f554e7efc41456: Status 404 returned error can't find the container with id 59d438d277816140b3b45ad12600d8901889c71785dcdf0843f554e7efc41456 Sep 15 21:23:01.097565 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.097537 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:23:01.097623 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.097601 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:23:01.579497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.579458 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" event={"ID":"53015761-220d-4d64-8b2f-d317680d9ffa","Type":"ContainerStarted","Data":"c7c32681b8813fdff9a35f69bc63dc5ea0e351f44991589a24b636390ae140df"} Sep 15 21:23:01.579915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.579519 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" event={"ID":"53015761-220d-4d64-8b2f-d317680d9ffa","Type":"ContainerStarted","Data":"59d438d277816140b3b45ad12600d8901889c71785dcdf0843f554e7efc41456"} Sep 15 21:23:01.579915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:01.579544 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:02.586440 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.586409 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" Sep 15 21:23:02.622911 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.622857 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5s4kc" podStartSLOduration=2.622840932 podStartE2EDuration="2.622840932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:01.612797258 +0000 UTC m=+422.247949370" watchObservedRunningTime="2026-09-15 21:23:02.622840932 +0000 UTC m=+423.257993035" Sep 15 21:23:02.670948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.670918 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:23:02.671156 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.671131 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" containerName="discovery" containerID="cri-o://17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71" gracePeriod=30 Sep 15 21:23:02.906215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.906186 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:23:02.960160 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960130 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960177 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q7f6\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960231 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960257 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960300 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960324 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960361 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs\") pod \"c66c738c-01ab-4380-94a2-385ebc63633b\" (UID: \"c66c738c-01ab-4380-94a2-385ebc63633b\") " Sep 15 21:23:02.960636 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.960591 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:23:02.963227 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963196 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs" (OuterVolumeSpecName: "local-certs") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:23:02.963366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963277 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:23:02.963366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963287 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:23:02.963366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963301 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts" (OuterVolumeSpecName: "cacerts") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:23:02.963603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963582 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:23:02.963795 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:02.963780 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6" (OuterVolumeSpecName: "kube-api-access-5q7f6") pod "c66c738c-01ab-4380-94a2-385ebc63633b" (UID: "c66c738c-01ab-4380-94a2-385ebc63633b"). InnerVolumeSpecName "kube-api-access-5q7f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:23:03.061685 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061646 2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-kubeconfig\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061685 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061685 2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-cacerts\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061685 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061698 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-dns-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061710 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061723 2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66c738c-01ab-4380-94a2-385ebc63633b-local-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061737 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66c738c-01ab-4380-94a2-385ebc63633b-istio-csr-ca-configmap\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.061960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.061750 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7f6\" (UniqueName: \"kubernetes.io/projected/c66c738c-01ab-4380-94a2-385ebc63633b-kube-api-access-5q7f6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:03.591029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.590993 2552 generic.go:356] "Generic (PLEG): container finished" podID="c66c738c-01ab-4380-94a2-385ebc63633b" containerID="17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71" exitCode=0 Sep 15 21:23:03.591424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.591044 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" Sep 15 21:23:03.591424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.591085 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" event={"ID":"c66c738c-01ab-4380-94a2-385ebc63633b","Type":"ContainerDied","Data":"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71"} Sep 15 21:23:03.591424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.591140 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f" event={"ID":"c66c738c-01ab-4380-94a2-385ebc63633b","Type":"ContainerDied","Data":"b41378512a7dda6ebe86d13d63e5649cdfb64686d2803f16e7c93fafde88f314"} Sep 15 21:23:03.591424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.591161 2552 scope.go:117] "RemoveContainer" containerID="17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71" Sep 15 21:23:03.599650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.599630 2552 scope.go:117] "RemoveContainer" containerID="17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71" Sep 15 21:23:03.599930 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:23:03.599910 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71\": container with ID starting with 17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71 not found: ID does not exist" containerID="17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71" Sep 15 21:23:03.599982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.599939 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71"} err="failed to get container status \"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71\": rpc error: code = NotFound desc = could not find container \"17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71\": container with ID starting with 17747cee745188229d1b21b567f086a08377a6dad4dcce5b344dbc02ba027e71 not found: ID does not exist" Sep 15 21:23:03.643563 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.643529 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:23:03.658108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.658054 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9h76f"] Sep 15 21:23:03.790034 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:03.789999 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" path="/var/lib/kubelet/pods/c66c738c-01ab-4380-94a2-385ebc63633b/volumes" Sep 15 21:23:04.057870 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.057820 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb"] Sep 15 21:23:04.058218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.058201 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" containerName="discovery" Sep 15 21:23:04.058319 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.058222 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" containerName="discovery" Sep 15 21:23:04.058373 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.058329 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66c738c-01ab-4380-94a2-385ebc63633b" containerName="discovery" Sep 15 21:23:04.062580 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.062557 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.066513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.066460 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 15 21:23:04.066642 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.066590 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lfvcn\"" Sep 15 21:23:04.081325 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.081301 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb"] Sep 15 21:23:04.171921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.171883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pb7z\" (UniqueName: \"kubernetes.io/projected/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-kube-api-access-7pb7z\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.172065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.172003 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.273431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.273398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.273606 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.273473 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb7z\" (UniqueName: \"kubernetes.io/projected/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-kube-api-access-7pb7z\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.273606 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:23:04.273554 2552 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 15 21:23:04.273682 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:23:04.273622 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert podName:9bfeed15-c9e3-43da-bb00-8f6e0554d67e nodeName:}" failed. No retries permitted until 2026-09-15 21:23:04.773605131 +0000 UTC m=+425.408757220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert") pod "kserve-controller-manager-d6f4b65b8-m9jtb" (UID: "9bfeed15-c9e3-43da-bb00-8f6e0554d67e") : secret "kserve-webhook-server-cert" not found Sep 15 21:23:04.288152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.288125 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pb7z\" (UniqueName: \"kubernetes.io/projected/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-kube-api-access-7pb7z\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.778661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.778612 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.781219 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.781193 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfeed15-c9e3-43da-bb00-8f6e0554d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-m9jtb\" (UID: \"9bfeed15-c9e3-43da-bb00-8f6e0554d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:04.973339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:04.973301 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:05.254176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:05.254142 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb"] Sep 15 21:23:05.257472 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:23:05.257438 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfeed15_c9e3_43da_bb00_8f6e0554d67e.slice/crio-525dbc452e7929d17b7c08b4d5a0e3ea59879165cecc15af3228025f58533a57 WatchSource:0}: Error finding container 525dbc452e7929d17b7c08b4d5a0e3ea59879165cecc15af3228025f58533a57: Status 404 returned error can't find the container with id 525dbc452e7929d17b7c08b4d5a0e3ea59879165cecc15af3228025f58533a57 Sep 15 21:23:05.608631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:05.608541 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" event={"ID":"9bfeed15-c9e3-43da-bb00-8f6e0554d67e","Type":"ContainerStarted","Data":"525dbc452e7929d17b7c08b4d5a0e3ea59879165cecc15af3228025f58533a57"} Sep 15 21:23:08.624803 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:08.624769 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" event={"ID":"9bfeed15-c9e3-43da-bb00-8f6e0554d67e","Type":"ContainerStarted","Data":"5ebb22833f07cc1c2503127e879c92d40154ec6731001598cced97c8697cf8e9"} Sep 15 21:23:08.625260 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:08.624898 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:08.645519 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:08.645464 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" podStartSLOduration=2.275064998 podStartE2EDuration="4.645448175s" totalImagesPullingTime="2.370383177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:08.64449916 +0000 UTC m=+429.279651296" watchObservedRunningTime="2026-09-15 21:23:08.645448175 +0000 UTC m=+429.280600286" Sep 15 21:23:14.614346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.614283 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d7c8859fd-m2d2j" podUID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" containerName="console" containerID="cri-o://dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f" gracePeriod=15 Sep 15 21:23:14.862433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.862409 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:23:14.969535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969508 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.969702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969549 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.969702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969586 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.969702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969641 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.969702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969685 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969730 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.969756 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-727zg\" (UniqueName: \"kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg\") pod \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\" (UID: \"fd1833dc-cd66-4adb-94c9-e7af666b2f8e\") " Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.970051 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config" (OuterVolumeSpecName: "console-config") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.970003 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca" (OuterVolumeSpecName: "service-ca") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.970158 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:23:14.970354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.970152 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:23:14.971893 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.971871 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg" (OuterVolumeSpecName: "kube-api-access-727zg") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "kube-api-access-727zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:23:14.971997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.971944 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:23:14.972056 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:14.971989 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fd1833dc-cd66-4adb-94c9-e7af666b2f8e" (UID: "fd1833dc-cd66-4adb-94c9-e7af666b2f8e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:23:15.070992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.070959 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-oauth-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.070992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.070986 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-oauth-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.070992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.070995 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-727zg\" (UniqueName: \"kubernetes.io/projected/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-kube-api-access-727zg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.071200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.071006 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.071200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.071015 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-config\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.071200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.071023 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-trusted-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.071200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.071031 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1833dc-cd66-4adb-94c9-e7af666b2f8e-console-serving-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:23:15.659187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.659148 2552 generic.go:356] "Generic (PLEG): container finished" podID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" containerID="dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f" exitCode=0 Sep 15 21:23:15.659636 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.659195 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c8859fd-m2d2j" Sep 15 21:23:15.659636 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.659241 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c8859fd-m2d2j" event={"ID":"fd1833dc-cd66-4adb-94c9-e7af666b2f8e","Type":"ContainerDied","Data":"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f"} Sep 15 21:23:15.659636 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.659294 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c8859fd-m2d2j" event={"ID":"fd1833dc-cd66-4adb-94c9-e7af666b2f8e","Type":"ContainerDied","Data":"c12744f7011e68faa5bdee39d9f54923c59f421d35ddc948a672ad70a3ab317c"} Sep 15 21:23:15.659636 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.659316 2552 scope.go:117] "RemoveContainer" containerID="dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f" Sep 15 21:23:15.667881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.667863 2552 scope.go:117] "RemoveContainer" containerID="dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f" Sep 15 21:23:15.668163 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:23:15.668145 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f\": container with ID starting with dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f not found: ID does not exist" containerID="dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f" Sep 15 21:23:15.668210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.668172 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f"} err="failed to get container status \"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f\": rpc error: code = NotFound desc = could not find container \"dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f\": container with ID starting with dd4d9e7a7b8da5d17929ee3fe97003a57503fde6c9d7a97335c9d871504fa65f not found: ID does not exist" Sep 15 21:23:15.682961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.682939 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:23:15.686721 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.686700 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d7c8859fd-m2d2j"] Sep 15 21:23:15.790144 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:15.790110 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" path="/var/lib/kubelet/pods/fd1833dc-cd66-4adb-94c9-e7af666b2f8e/volumes" Sep 15 21:23:39.635303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:39.635271 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m9jtb" Sep 15 21:23:59.985467 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.985435 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5"] Sep 15 21:23:59.985782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.985710 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" containerName="console" Sep 15 21:23:59.985782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.985722 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" containerName="console" Sep 15 21:23:59.985782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.985780 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1833dc-cd66-4adb-94c9-e7af666b2f8e" containerName="console" Sep 15 21:23:59.989935 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.989918 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:23:59.992712 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.992690 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 21:23:59.992815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.992697 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 21:23:59.992815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.992785 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zznjp\"" Sep 15 21:23:59.998688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:23:59.998666 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5"] Sep 15 21:24:00.063630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.063552 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn"] Sep 15 21:24:00.066762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.066746 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.076873 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.076849 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn"] Sep 15 21:24:00.116698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116672 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.116875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116702 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhcm\" (UniqueName: \"kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.116875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116724 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.116875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.116875 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116846 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjkv\" (UniqueName: \"kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.117035 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.116897 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.164303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.164276 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6"] Sep 15 21:24:00.167596 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.167578 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.176425 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.176404 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6"] Sep 15 21:24:00.217972 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.217947 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.218084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.217973 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhcm\" (UniqueName: \"kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.218084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.217992 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.218084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218011 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.218084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218042 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.218296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218097 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjkv\" (UniqueName: \"kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.218296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218146 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.218296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxgf\" (UniqueName: \"kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.218296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218235 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.218461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218401 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.218461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218419 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.218461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218401 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.218461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.218450 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.226943 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.226915 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhcm\" (UniqueName: \"kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.226943 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.226936 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjkv\" (UniqueName: \"kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.260486 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.260466 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg"] Sep 15 21:24:00.264474 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.264458 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.274855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.274819 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg"] Sep 15 21:24:00.299648 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.299623 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:00.319390 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319334 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.319483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319406 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxgf\" (UniqueName: \"kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.319483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319433 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntr7\" (UniqueName: \"kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.319483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319467 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.319668 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319487 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.319668 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.319742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319681 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.319938 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.319918 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.330115 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.330094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxgf\" (UniqueName: \"kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.375879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.375807 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:00.420675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.420631 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntr7\" (UniqueName: \"kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.420821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.420695 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.420821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.420721 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.421125 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.421108 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.421543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.421526 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.433808 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.433754 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntr7\" (UniqueName: \"kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.477462 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.477438 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:00.572305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.572281 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5"] Sep 15 21:24:00.573698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.573685 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:00.574009 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:00.573987 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e33eaa_d4a5_4674_bfa8_2dd06ad182cc.slice/crio-592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf WatchSource:0}: Error finding container 592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf: Status 404 returned error can't find the container with id 592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf Sep 15 21:24:00.655744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.655722 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn"] Sep 15 21:24:00.658679 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:00.658654 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396b0637_a385_4c2e_b8df_6f344456758f.slice/crio-5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651 WatchSource:0}: Error finding container 5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651: Status 404 returned error can't find the container with id 5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651 Sep 15 21:24:00.757718 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:00.757681 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43b8ab0_207d_449c_b7f9_6019d6852f3d.slice/crio-fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841 WatchSource:0}: Error finding container fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841: Status 404 returned error can't find the container with id fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841 Sep 15 21:24:00.759168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.759132 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6"] Sep 15 21:24:00.847046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.846996 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg"] Sep 15 21:24:00.849556 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:00.849533 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc200d786_cee8_4406_831e_d0bf0cec6df0.slice/crio-ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7 WatchSource:0}: Error finding container ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7: Status 404 returned error can't find the container with id ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7 Sep 15 21:24:00.878206 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.878176 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" event={"ID":"c200d786-cee8-4406-831e-d0bf0cec6df0","Type":"ContainerStarted","Data":"ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7"} Sep 15 21:24:00.879881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.879854 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerStarted","Data":"5213ed259397eb033330008be486a969994aa956ae7281be44fe2c7c6323658e"} Sep 15 21:24:00.879989 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.879909 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerStarted","Data":"fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841"} Sep 15 21:24:00.881627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.881603 2552 generic.go:356] "Generic (PLEG): container finished" podID="396b0637-a385-4c2e-b8df-6f344456758f" containerID="895967621acce24b965a19aecc729023095487be687a5e2f1b5447510123092d" exitCode=0 Sep 15 21:24:00.881732 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.881682 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" event={"ID":"396b0637-a385-4c2e-b8df-6f344456758f","Type":"ContainerDied","Data":"895967621acce24b965a19aecc729023095487be687a5e2f1b5447510123092d"} Sep 15 21:24:00.881796 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.881743 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" event={"ID":"396b0637-a385-4c2e-b8df-6f344456758f","Type":"ContainerStarted","Data":"5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651"} Sep 15 21:24:00.883551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.883511 2552 generic.go:356] "Generic (PLEG): container finished" podID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerID="074f8136bf909e7b6548bc4b9a93c576e995399a2ef0572a947a1b8922c00844" exitCode=0 Sep 15 21:24:00.883551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.883537 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" event={"ID":"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc","Type":"ContainerDied","Data":"074f8136bf909e7b6548bc4b9a93c576e995399a2ef0572a947a1b8922c00844"} Sep 15 21:24:00.883665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:00.883576 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" event={"ID":"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc","Type":"ContainerStarted","Data":"592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf"} Sep 15 21:24:01.889882 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.889851 2552 generic.go:356] "Generic (PLEG): container finished" podID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerID="79e8c973fc839b60ed05183cfcf9dd0998d616a504b2c99961982322fdca1f9c" exitCode=0 Sep 15 21:24:01.890322 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.889925 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" event={"ID":"c200d786-cee8-4406-831e-d0bf0cec6df0","Type":"ContainerDied","Data":"79e8c973fc839b60ed05183cfcf9dd0998d616a504b2c99961982322fdca1f9c"} Sep 15 21:24:01.891403 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.891382 2552 generic.go:356] "Generic (PLEG): container finished" podID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerID="5213ed259397eb033330008be486a969994aa956ae7281be44fe2c7c6323658e" exitCode=0 Sep 15 21:24:01.891502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.891455 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerDied","Data":"5213ed259397eb033330008be486a969994aa956ae7281be44fe2c7c6323658e"} Sep 15 21:24:01.893545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.893524 2552 generic.go:356] "Generic (PLEG): container finished" podID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerID="de02e178fccb8932bc184e0c25e1ba58e06468ffd26cf99a9f7bec18d76bfa5e" exitCode=0 Sep 15 21:24:01.893545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:01.893553 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" event={"ID":"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc","Type":"ContainerDied","Data":"de02e178fccb8932bc184e0c25e1ba58e06468ffd26cf99a9f7bec18d76bfa5e"} Sep 15 21:24:02.901016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:02.900986 2552 generic.go:356] "Generic (PLEG): container finished" podID="396b0637-a385-4c2e-b8df-6f344456758f" containerID="75e9e8cc227a9a0ff3a3620a18258bd181227f5e6213a31a1703965fe0c85253" exitCode=0 Sep 15 21:24:02.901528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:02.901070 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" event={"ID":"396b0637-a385-4c2e-b8df-6f344456758f","Type":"ContainerDied","Data":"75e9e8cc227a9a0ff3a3620a18258bd181227f5e6213a31a1703965fe0c85253"} Sep 15 21:24:02.903250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:02.903229 2552 generic.go:356] "Generic (PLEG): container finished" podID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerID="dc0a40307e8b207b9d048d2ae3b7c4c227af54abf21d80f23597033cdce29e83" exitCode=0 Sep 15 21:24:02.903318 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:02.903271 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" event={"ID":"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc","Type":"ContainerDied","Data":"dc0a40307e8b207b9d048d2ae3b7c4c227af54abf21d80f23597033cdce29e83"} Sep 15 21:24:03.910418 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.910385 2552 generic.go:356] "Generic (PLEG): container finished" podID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerID="1b4210d2495e9151ebdc6439daad57f5f7b027637a1267a86920a713d379cfce" exitCode=0 Sep 15 21:24:03.910881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.910463 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" event={"ID":"c200d786-cee8-4406-831e-d0bf0cec6df0","Type":"ContainerDied","Data":"1b4210d2495e9151ebdc6439daad57f5f7b027637a1267a86920a713d379cfce"} Sep 15 21:24:03.912435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.912410 2552 generic.go:356] "Generic (PLEG): container finished" podID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerID="8bc99c1a0ef3ff7d1cf44c6347e2b545607c7b0310a4bdbe316caea1c10d37a7" exitCode=0 Sep 15 21:24:03.912532 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.912483 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerDied","Data":"8bc99c1a0ef3ff7d1cf44c6347e2b545607c7b0310a4bdbe316caea1c10d37a7"} Sep 15 21:24:03.914757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.914734 2552 generic.go:356] "Generic (PLEG): container finished" podID="396b0637-a385-4c2e-b8df-6f344456758f" containerID="774604c7a4c62c1ac696044ab1b8794c22897c460b55f007a030db39fbab7bf7" exitCode=0 Sep 15 21:24:03.914837 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:03.914809 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" event={"ID":"396b0637-a385-4c2e-b8df-6f344456758f","Type":"ContainerDied","Data":"774604c7a4c62c1ac696044ab1b8794c22897c460b55f007a030db39fbab7bf7"} Sep 15 21:24:04.051047 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.051023 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:04.149284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.149255 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbhcm\" (UniqueName: \"kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm\") pod \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " Sep 15 21:24:04.149396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.149309 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle\") pod \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " Sep 15 21:24:04.149396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.149345 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util\") pod \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\" (UID: \"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc\") " Sep 15 21:24:04.150052 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.150019 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle" (OuterVolumeSpecName: "bundle") pod "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" (UID: "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:04.151266 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.151245 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm" (OuterVolumeSpecName: "kube-api-access-lbhcm") pod "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" (UID: "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc"). InnerVolumeSpecName "kube-api-access-lbhcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:04.154429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.154398 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util" (OuterVolumeSpecName: "util") pod "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" (UID: "57e33eaa-d4a5-4674-bfa8-2dd06ad182cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:04.250615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.250589 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:04.250615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.250614 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbhcm\" (UniqueName: \"kubernetes.io/projected/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-kube-api-access-lbhcm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:04.250754 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.250626 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57e33eaa-d4a5-4674-bfa8-2dd06ad182cc-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:04.923450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.923415 2552 generic.go:356] "Generic (PLEG): container finished" podID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerID="e75efdce97564d6c9566b4ebd39ff6914ef5d9cc867ff244e49fdcef337aa8d3" exitCode=0 Sep 15 21:24:04.923899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.923498 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" event={"ID":"c200d786-cee8-4406-831e-d0bf0cec6df0","Type":"ContainerDied","Data":"e75efdce97564d6c9566b4ebd39ff6914ef5d9cc867ff244e49fdcef337aa8d3"} Sep 15 21:24:04.925733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.925705 2552 generic.go:356] "Generic (PLEG): container finished" podID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerID="e43fc96cd95b5cd61f6e1660f7f87d52e09f5f7ea373ccd7a2ee9851217e4c65" exitCode=0 Sep 15 21:24:04.925898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.925803 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerDied","Data":"e43fc96cd95b5cd61f6e1660f7f87d52e09f5f7ea373ccd7a2ee9851217e4c65"} Sep 15 21:24:04.927684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.927664 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" event={"ID":"57e33eaa-d4a5-4674-bfa8-2dd06ad182cc","Type":"ContainerDied","Data":"592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf"} Sep 15 21:24:04.927769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.927683 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak8gr5" Sep 15 21:24:04.927769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:04.927708 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592dfd010e0c4a180cc482ae95a012cd0968780bd25ec5c5ff6d38bda8b917bf" Sep 15 21:24:05.054309 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.054285 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:05.158090 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.158056 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgjkv\" (UniqueName: \"kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv\") pod \"396b0637-a385-4c2e-b8df-6f344456758f\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " Sep 15 21:24:05.158258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.158102 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle\") pod \"396b0637-a385-4c2e-b8df-6f344456758f\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " Sep 15 21:24:05.158258 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.158133 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util\") pod \"396b0637-a385-4c2e-b8df-6f344456758f\" (UID: \"396b0637-a385-4c2e-b8df-6f344456758f\") " Sep 15 21:24:05.158632 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.158602 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle" (OuterVolumeSpecName: "bundle") pod "396b0637-a385-4c2e-b8df-6f344456758f" (UID: "396b0637-a385-4c2e-b8df-6f344456758f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:05.160195 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.160172 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv" (OuterVolumeSpecName: "kube-api-access-dgjkv") pod "396b0637-a385-4c2e-b8df-6f344456758f" (UID: "396b0637-a385-4c2e-b8df-6f344456758f"). InnerVolumeSpecName "kube-api-access-dgjkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:05.163313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.163293 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util" (OuterVolumeSpecName: "util") pod "396b0637-a385-4c2e-b8df-6f344456758f" (UID: "396b0637-a385-4c2e-b8df-6f344456758f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:05.258769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.258688 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgjkv\" (UniqueName: \"kubernetes.io/projected/396b0637-a385-4c2e-b8df-6f344456758f-kube-api-access-dgjkv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:05.258769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.258719 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:05.258769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.258729 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396b0637-a385-4c2e-b8df-6f344456758f-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:05.934670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.934619 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" event={"ID":"396b0637-a385-4c2e-b8df-6f344456758f","Type":"ContainerDied","Data":"5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651"} Sep 15 21:24:05.935084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.934683 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d9b920a5959e1f1320bcb6db31a26a0f9ce134a4a3e2532bbe5eaa15f8c6651" Sep 15 21:24:05.935084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:05.934640 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zpbxn" Sep 15 21:24:06.086577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.086556 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:06.088588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.088570 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:06.166576 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166549 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle\") pod \"c200d786-cee8-4406-831e-d0bf0cec6df0\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " Sep 15 21:24:06.166763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166597 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpxgf\" (UniqueName: \"kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf\") pod \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " Sep 15 21:24:06.166763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166638 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle\") pod \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " Sep 15 21:24:06.166763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166674 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util\") pod \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\" (UID: \"b43b8ab0-207d-449c-b7f9-6019d6852f3d\") " Sep 15 21:24:06.166763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166701 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ntr7\" (UniqueName: \"kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7\") pod \"c200d786-cee8-4406-831e-d0bf0cec6df0\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " Sep 15 21:24:06.166998 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.166784 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util\") pod \"c200d786-cee8-4406-831e-d0bf0cec6df0\" (UID: \"c200d786-cee8-4406-831e-d0bf0cec6df0\") " Sep 15 21:24:06.167318 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.167288 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle" (OuterVolumeSpecName: "bundle") pod "b43b8ab0-207d-449c-b7f9-6019d6852f3d" (UID: "b43b8ab0-207d-449c-b7f9-6019d6852f3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:06.167622 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.167594 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle" (OuterVolumeSpecName: "bundle") pod "c200d786-cee8-4406-831e-d0bf0cec6df0" (UID: "c200d786-cee8-4406-831e-d0bf0cec6df0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:06.168891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.168861 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7" (OuterVolumeSpecName: "kube-api-access-6ntr7") pod "c200d786-cee8-4406-831e-d0bf0cec6df0" (UID: "c200d786-cee8-4406-831e-d0bf0cec6df0"). InnerVolumeSpecName "kube-api-access-6ntr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:06.168977 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.168908 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf" (OuterVolumeSpecName: "kube-api-access-lpxgf") pod "b43b8ab0-207d-449c-b7f9-6019d6852f3d" (UID: "b43b8ab0-207d-449c-b7f9-6019d6852f3d"). InnerVolumeSpecName "kube-api-access-lpxgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:06.172437 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.172405 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util" (OuterVolumeSpecName: "util") pod "c200d786-cee8-4406-831e-d0bf0cec6df0" (UID: "c200d786-cee8-4406-831e-d0bf0cec6df0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:06.174376 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.174355 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util" (OuterVolumeSpecName: "util") pod "b43b8ab0-207d-449c-b7f9-6019d6852f3d" (UID: "b43b8ab0-207d-449c-b7f9-6019d6852f3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:06.267599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267563 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.267599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267597 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntr7\" (UniqueName: \"kubernetes.io/projected/c200d786-cee8-4406-831e-d0bf0cec6df0-kube-api-access-6ntr7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.267599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267607 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-util\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.267769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267616 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c200d786-cee8-4406-831e-d0bf0cec6df0-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.267769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267624 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxgf\" (UniqueName: \"kubernetes.io/projected/b43b8ab0-207d-449c-b7f9-6019d6852f3d-kube-api-access-lpxgf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.267769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.267632 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b43b8ab0-207d-449c-b7f9-6019d6852f3d-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:06.941652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.941624 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" Sep 15 21:24:06.942068 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.941620 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nr7qg" event={"ID":"c200d786-cee8-4406-831e-d0bf0cec6df0","Type":"ContainerDied","Data":"ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7"} Sep 15 21:24:06.942068 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.941774 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4958af7328d5ee53a5d68d67e7bfb945b9b44957041c76b566fce36b80cef7" Sep 15 21:24:06.943555 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.943529 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" event={"ID":"b43b8ab0-207d-449c-b7f9-6019d6852f3d","Type":"ContainerDied","Data":"fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841"} Sep 15 21:24:06.943641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.943557 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czh2n6" Sep 15 21:24:06.943641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:06.943568 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc87b11e0644c74fae807821ac353b72d3e5bfb1536238692a14bc4083e52841" Sep 15 21:24:15.045747 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.045712 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8"] Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046032 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046045 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046053 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046060 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046067 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046073 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046080 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046086 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046092 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046097 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046104 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046110 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="extract" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046123 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046130 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046139 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046146 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046156 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046160 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046166 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046173 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046180 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046184 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="pull" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046191 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046196 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="util" Sep 15 21:24:15.046234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046248 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="396b0637-a385-4c2e-b8df-6f344456758f" containerName="extract" Sep 15 21:24:15.046994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046256 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43b8ab0-207d-449c-b7f9-6019d6852f3d" containerName="extract" Sep 15 21:24:15.046994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046263 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c200d786-cee8-4406-831e-d0bf0cec6df0" containerName="extract" Sep 15 21:24:15.046994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.046269 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e33eaa-d4a5-4674-bfa8-2dd06ad182cc" containerName="extract" Sep 15 21:24:15.050686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.050671 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:15.053651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.053629 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4xcjj\"" Sep 15 21:24:15.054291 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.054275 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 21:24:15.054371 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.054346 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 21:24:15.065324 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.065301 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8"] Sep 15 21:24:15.146689 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.146662 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnvb\" (UniqueName: \"kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb\") pod \"limitador-operator-controller-manager-74495c69f8-74pk8\" (UID: \"4c1e23a5-3561-4806-b22c-4268390a6c12\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:15.247392 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.247359 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnvb\" (UniqueName: \"kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb\") pod \"limitador-operator-controller-manager-74495c69f8-74pk8\" (UID: \"4c1e23a5-3561-4806-b22c-4268390a6c12\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:15.274063 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.274037 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnvb\" (UniqueName: \"kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb\") pod \"limitador-operator-controller-manager-74495c69f8-74pk8\" (UID: \"4c1e23a5-3561-4806-b22c-4268390a6c12\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:15.360873 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.360790 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:15.636754 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.636732 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8"] Sep 15 21:24:15.638427 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:15.638402 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1e23a5_3561_4806_b22c_4268390a6c12.slice/crio-20babca3aec706f25c56ba5fb75d39a6d1dd97a2b41a8c1ca55377fc31d4f3c9 WatchSource:0}: Error finding container 20babca3aec706f25c56ba5fb75d39a6d1dd97a2b41a8c1ca55377fc31d4f3c9: Status 404 returned error can't find the container with id 20babca3aec706f25c56ba5fb75d39a6d1dd97a2b41a8c1ca55377fc31d4f3c9 Sep 15 21:24:15.986943 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:15.986862 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" event={"ID":"4c1e23a5-3561-4806-b22c-4268390a6c12","Type":"ContainerStarted","Data":"20babca3aec706f25c56ba5fb75d39a6d1dd97a2b41a8c1ca55377fc31d4f3c9"} Sep 15 21:24:19.006061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:19.006020 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" event={"ID":"4c1e23a5-3561-4806-b22c-4268390a6c12","Type":"ContainerStarted","Data":"1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c"} Sep 15 21:24:19.006427 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:19.006139 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:19.047294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:19.047211 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" podStartSLOduration=1.2347342000000001 podStartE2EDuration="4.047195446s" totalImagesPullingTime="2.812461246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:19.045903551 +0000 UTC m=+499.681055662" watchObservedRunningTime="2026-09-15 21:24:19.047195446 +0000 UTC m=+499.682347556" Sep 15 21:24:22.329262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.329230 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-s9hnr"] Sep 15 21:24:22.332381 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.332358 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:22.334853 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.334817 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zw6x7\"" Sep 15 21:24:22.352352 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.352330 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-s9hnr"] Sep 15 21:24:22.512106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.512078 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxlh\" (UniqueName: \"kubernetes.io/projected/fb65d59f-23b3-43d3-9634-c5b9193c45a3-kube-api-access-qzxlh\") pod \"authorino-operator-76b478bd78-s9hnr\" (UID: \"fb65d59f-23b3-43d3-9634-c5b9193c45a3\") " pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:22.613476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.613397 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxlh\" (UniqueName: \"kubernetes.io/projected/fb65d59f-23b3-43d3-9634-c5b9193c45a3-kube-api-access-qzxlh\") pod \"authorino-operator-76b478bd78-s9hnr\" (UID: \"fb65d59f-23b3-43d3-9634-c5b9193c45a3\") " pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:22.622025 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.621995 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxlh\" (UniqueName: \"kubernetes.io/projected/fb65d59f-23b3-43d3-9634-c5b9193c45a3-kube-api-access-qzxlh\") pod \"authorino-operator-76b478bd78-s9hnr\" (UID: \"fb65d59f-23b3-43d3-9634-c5b9193c45a3\") " pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:22.641814 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.641790 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:22.911818 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:22.911794 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-s9hnr"] Sep 15 21:24:22.914064 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:22.914027 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb65d59f_23b3_43d3_9634_c5b9193c45a3.slice/crio-577b7c1d59f2ad94c6e439f8419930f8a5cd8564b744700982b8c65dd4c02875 WatchSource:0}: Error finding container 577b7c1d59f2ad94c6e439f8419930f8a5cd8564b744700982b8c65dd4c02875: Status 404 returned error can't find the container with id 577b7c1d59f2ad94c6e439f8419930f8a5cd8564b744700982b8c65dd4c02875 Sep 15 21:24:23.027101 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:23.026967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" event={"ID":"fb65d59f-23b3-43d3-9634-c5b9193c45a3","Type":"ContainerStarted","Data":"577b7c1d59f2ad94c6e439f8419930f8a5cd8564b744700982b8c65dd4c02875"} Sep 15 21:24:25.040315 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:25.040278 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" event={"ID":"fb65d59f-23b3-43d3-9634-c5b9193c45a3","Type":"ContainerStarted","Data":"8e855cef4e7fb14713c26b3621aba413fb4c74cb5f2d33cedf8962c0e501c1a2"} Sep 15 21:24:25.040706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:25.040395 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:25.058607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:25.058557 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" podStartSLOduration=1.582884023 podStartE2EDuration="3.058542253s" totalImagesPullingTime="1.47565823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:25.056897552 +0000 UTC m=+505.692049664" watchObservedRunningTime="2026-09-15 21:24:25.058542253 +0000 UTC m=+505.693694365" Sep 15 21:24:30.012807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:30.012780 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:36.047894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:36.047866 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-s9hnr" Sep 15 21:24:38.756485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.756451 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8"] Sep 15 21:24:38.756897 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.756692 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" containerName="manager" containerID="cri-o://1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c" gracePeriod=2 Sep 15 21:24:38.777762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.777733 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8"] Sep 15 21:24:38.790341 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:38.790292 2552 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4c1e23a5-3561-4806-b22c-4268390a6c12\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-74pk8\": pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:38.808439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.808411 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v"] Sep 15 21:24:38.808942 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.808925 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" containerName="manager" Sep 15 21:24:38.809002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.808946 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" containerName="manager" Sep 15 21:24:38.809044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.809034 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" containerName="manager" Sep 15 21:24:38.812350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.812332 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:38.814377 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:38.814346 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:38.830005 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.829983 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v"] Sep 15 21:24:38.835031 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.835011 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrrm\" (UniqueName: \"kubernetes.io/projected/220af762-58bc-4141-8776-57ebfd41fd65-kube-api-access-qdrrm\") pod \"limitador-operator-controller-manager-74495c69f8-vkc2v\" (UID: \"220af762-58bc-4141-8776-57ebfd41fd65\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:38.936465 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.936379 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrrm\" (UniqueName: \"kubernetes.io/projected/220af762-58bc-4141-8776-57ebfd41fd65-kube-api-access-qdrrm\") pod \"limitador-operator-controller-manager-74495c69f8-vkc2v\" (UID: \"220af762-58bc-4141-8776-57ebfd41fd65\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:38.945962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.945937 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrrm\" (UniqueName: \"kubernetes.io/projected/220af762-58bc-4141-8776-57ebfd41fd65-kube-api-access-qdrrm\") pod \"limitador-operator-controller-manager-74495c69f8-vkc2v\" (UID: \"220af762-58bc-4141-8776-57ebfd41fd65\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:38.995739 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:38.995719 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:38.997708 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:38.997677 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:39.037374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.037309 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gnvb\" (UniqueName: \"kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb\") pod \"4c1e23a5-3561-4806-b22c-4268390a6c12\" (UID: \"4c1e23a5-3561-4806-b22c-4268390a6c12\") " Sep 15 21:24:39.039411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.039378 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb" (OuterVolumeSpecName: "kube-api-access-9gnvb") pod "4c1e23a5-3561-4806-b22c-4268390a6c12" (UID: "4c1e23a5-3561-4806-b22c-4268390a6c12"). InnerVolumeSpecName "kube-api-access-9gnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:39.109289 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.109261 2552 generic.go:356] "Generic (PLEG): container finished" podID="4c1e23a5-3561-4806-b22c-4268390a6c12" containerID="1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c" exitCode=0 Sep 15 21:24:39.109419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.109308 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:39.109419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.109367 2552 scope.go:117] "RemoveContainer" containerID="1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c" Sep 15 21:24:39.111617 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:39.111586 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:39.118487 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.118472 2552 scope.go:117] "RemoveContainer" containerID="1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c" Sep 15 21:24:39.118761 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:39.118743 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c\": container with ID starting with 1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c not found: ID does not exist" containerID="1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c" Sep 15 21:24:39.118810 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.118769 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c"} err="failed to get container status \"1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c\": rpc error: code = NotFound desc = could not find container \"1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c\": container with ID starting with 1e1f8187d348436b62f23e228fb70afa8d15d64b00235994c20d92f96120879c not found: ID does not exist" Sep 15 21:24:39.120592 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:39.120572 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:39.127442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.127424 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:39.137760 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.137745 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnvb\" (UniqueName: \"kubernetes.io/projected/4c1e23a5-3561-4806-b22c-4268390a6c12-kube-api-access-9gnvb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:24:39.421730 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.421691 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v"] Sep 15 21:24:39.426018 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:24:39.425988 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220af762_58bc_4141_8776_57ebfd41fd65.slice/crio-84cd99bf9dbf769092a48ff6220087745f9a3bc557c998c946df4d0ea0ecfd86 WatchSource:0}: Error finding container 84cd99bf9dbf769092a48ff6220087745f9a3bc557c998c946df4d0ea0ecfd86: Status 404 returned error can't find the container with id 84cd99bf9dbf769092a48ff6220087745f9a3bc557c998c946df4d0ea0ecfd86 Sep 15 21:24:39.789791 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:39.789720 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" path="/var/lib/kubelet/pods/4c1e23a5-3561-4806-b22c-4268390a6c12/volumes" Sep 15 21:24:39.790768 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:24:39.790745 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74pk8\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="4c1e23a5-3561-4806-b22c-4268390a6c12" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74pk8" Sep 15 21:24:40.115890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:40.115856 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" event={"ID":"220af762-58bc-4141-8776-57ebfd41fd65","Type":"ContainerStarted","Data":"cd0bd7684369dee372a7bc16ddc7288eb10ad8750477b693c9b29ceaa2e920fb"} Sep 15 21:24:40.116069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:40.115914 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" event={"ID":"220af762-58bc-4141-8776-57ebfd41fd65","Type":"ContainerStarted","Data":"84cd99bf9dbf769092a48ff6220087745f9a3bc557c998c946df4d0ea0ecfd86"} Sep 15 21:24:40.116069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:40.115947 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:24:40.143186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:40.143136 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" podStartSLOduration=2.143119962 podStartE2EDuration="2.143119962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:40.142505801 +0000 UTC m=+520.777657913" watchObservedRunningTime="2026-09-15 21:24:40.143119962 +0000 UTC m=+520.778272074" Sep 15 21:24:51.124449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:24:51.124419 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vkc2v" Sep 15 21:25:25.158213 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.158179 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:25.161577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.161548 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:25.164400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.164377 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bdwpx\"" Sep 15 21:25:25.169633 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.169608 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:25.224615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.224576 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fkn\" (UniqueName: \"kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn\") pod \"authorino-7bcfd8f9cd-nw8hx\" (UID: \"bfcd9977-3d65-4239-98ea-b796e331e26a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:25.325273 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.325236 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fkn\" (UniqueName: \"kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn\") pod \"authorino-7bcfd8f9cd-nw8hx\" (UID: \"bfcd9977-3d65-4239-98ea-b796e331e26a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:25.333755 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.333723 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fkn\" (UniqueName: \"kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn\") pod \"authorino-7bcfd8f9cd-nw8hx\" (UID: \"bfcd9977-3d65-4239-98ea-b796e331e26a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:25.471520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.471431 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:25.740301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:25.740275 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:25.742725 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:25:25.742696 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfcd9977_3d65_4239_98ea_b796e331e26a.slice/crio-aa7a1fa86a89332c74fe877d2983be60d244f0d83ff88c263c4c7ec6021cfa52 WatchSource:0}: Error finding container aa7a1fa86a89332c74fe877d2983be60d244f0d83ff88c263c4c7ec6021cfa52: Status 404 returned error can't find the container with id aa7a1fa86a89332c74fe877d2983be60d244f0d83ff88c263c4c7ec6021cfa52 Sep 15 21:25:26.346722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:26.346686 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" event={"ID":"bfcd9977-3d65-4239-98ea-b796e331e26a","Type":"ContainerStarted","Data":"aa7a1fa86a89332c74fe877d2983be60d244f0d83ff88c263c4c7ec6021cfa52"} Sep 15 21:25:28.359205 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:28.359112 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" event={"ID":"bfcd9977-3d65-4239-98ea-b796e331e26a","Type":"ContainerStarted","Data":"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952"} Sep 15 21:25:28.376085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:28.376034 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" podStartSLOduration=1.079215757 podStartE2EDuration="3.376018333s" totalImagesPullingTime="2.296802576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:28.374563222 +0000 UTC m=+569.009715346" watchObservedRunningTime="2026-09-15 21:25:28.376018333 +0000 UTC m=+569.011170489" Sep 15 21:25:28.744154 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:28.744122 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:30.369372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:30.369334 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" podUID="bfcd9977-3d65-4239-98ea-b796e331e26a" containerName="authorino" containerID="cri-o://07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952" gracePeriod=30 Sep 15 21:25:30.613394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:30.613365 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:30.667197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:30.667116 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95fkn\" (UniqueName: \"kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn\") pod \"bfcd9977-3d65-4239-98ea-b796e331e26a\" (UID: \"bfcd9977-3d65-4239-98ea-b796e331e26a\") " Sep 15 21:25:30.669081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:30.669057 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn" (OuterVolumeSpecName: "kube-api-access-95fkn") pod "bfcd9977-3d65-4239-98ea-b796e331e26a" (UID: "bfcd9977-3d65-4239-98ea-b796e331e26a"). InnerVolumeSpecName "kube-api-access-95fkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:25:30.767804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:30.767769 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fkn\" (UniqueName: \"kubernetes.io/projected/bfcd9977-3d65-4239-98ea-b796e331e26a-kube-api-access-95fkn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:25:31.375270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.375231 2552 generic.go:356] "Generic (PLEG): container finished" podID="bfcd9977-3d65-4239-98ea-b796e331e26a" containerID="07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952" exitCode=0 Sep 15 21:25:31.375690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.375281 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" Sep 15 21:25:31.375690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.375318 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" event={"ID":"bfcd9977-3d65-4239-98ea-b796e331e26a","Type":"ContainerDied","Data":"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952"} Sep 15 21:25:31.375690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.375363 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nw8hx" event={"ID":"bfcd9977-3d65-4239-98ea-b796e331e26a","Type":"ContainerDied","Data":"aa7a1fa86a89332c74fe877d2983be60d244f0d83ff88c263c4c7ec6021cfa52"} Sep 15 21:25:31.375690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.375386 2552 scope.go:117] "RemoveContainer" containerID="07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952" Sep 15 21:25:31.385147 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.385130 2552 scope.go:117] "RemoveContainer" containerID="07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952" Sep 15 21:25:31.385414 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:25:31.385394 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952\": container with ID starting with 07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952 not found: ID does not exist" containerID="07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952" Sep 15 21:25:31.385478 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.385422 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952"} err="failed to get container status \"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952\": rpc error: code = NotFound desc = could not find container \"07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952\": container with ID starting with 07e284bd223f4b446ed2ce038491081bd53d1de1e3738a6899ebf145f0e7b952 not found: ID does not exist" Sep 15 21:25:31.401728 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.401697 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:31.412324 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.412303 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nw8hx"] Sep 15 21:25:31.790278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:31.790199 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfcd9977-3d65-4239-98ea-b796e331e26a" path="/var/lib/kubelet/pods/bfcd9977-3d65-4239-98ea-b796e331e26a/volumes" Sep 15 21:25:42.342360 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.342330 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-72dr8"] Sep 15 21:25:42.342800 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.342708 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcd9977-3d65-4239-98ea-b796e331e26a" containerName="authorino" Sep 15 21:25:42.342800 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.342724 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcd9977-3d65-4239-98ea-b796e331e26a" containerName="authorino" Sep 15 21:25:42.342927 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.342793 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcd9977-3d65-4239-98ea-b796e331e26a" containerName="authorino" Sep 15 21:25:42.347223 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.347206 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.349723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.349695 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 15 21:25:42.350361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.350338 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 21:25:42.350361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.350349 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 21:25:42.350665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.350650 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wzjgc\"" Sep 15 21:25:42.355412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.355392 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-72dr8"] Sep 15 21:25:42.466363 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.466328 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmqn\" (UniqueName: \"kubernetes.io/projected/84ec2d4f-f965-48ac-a26f-9897a17c71e8-kube-api-access-fgmqn\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.466552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.466477 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84ec2d4f-f965-48ac-a26f-9897a17c71e8-data\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.567559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.567509 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84ec2d4f-f965-48ac-a26f-9897a17c71e8-data\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.567761 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.567580 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmqn\" (UniqueName: \"kubernetes.io/projected/84ec2d4f-f965-48ac-a26f-9897a17c71e8-kube-api-access-fgmqn\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.567926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.567904 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84ec2d4f-f965-48ac-a26f-9897a17c71e8-data\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.586050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.586027 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmqn\" (UniqueName: \"kubernetes.io/projected/84ec2d4f-f965-48ac-a26f-9897a17c71e8-kube-api-access-fgmqn\") pod \"postgres-69fb4594cf-72dr8\" (UID: \"84ec2d4f-f965-48ac-a26f-9897a17c71e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.659219 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.659191 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:42.928633 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:42.928602 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-72dr8"] Sep 15 21:25:42.930656 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:25:42.930623 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ec2d4f_f965_48ac_a26f_9897a17c71e8.slice/crio-5ef01a4f3d59c3e107a4a0e4f6bb46b6015f6b6885d8202b44501b6e8c9c9770 WatchSource:0}: Error finding container 5ef01a4f3d59c3e107a4a0e4f6bb46b6015f6b6885d8202b44501b6e8c9c9770: Status 404 returned error can't find the container with id 5ef01a4f3d59c3e107a4a0e4f6bb46b6015f6b6885d8202b44501b6e8c9c9770 Sep 15 21:25:43.435977 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:43.435944 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" event={"ID":"84ec2d4f-f965-48ac-a26f-9897a17c71e8","Type":"ContainerStarted","Data":"5ef01a4f3d59c3e107a4a0e4f6bb46b6015f6b6885d8202b44501b6e8c9c9770"} Sep 15 21:25:48.467530 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:48.467492 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" event={"ID":"84ec2d4f-f965-48ac-a26f-9897a17c71e8","Type":"ContainerStarted","Data":"282e89834cdf8910a374475ac3a7e280b2f2d8d938d86e37f96cbe434a5350e2"} Sep 15 21:25:48.467929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:48.467626 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:48.504294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:48.504245 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" podStartSLOduration=1.503541093 podStartE2EDuration="6.504229212s" totalImagesPullingTime="5.000688119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:48.503147521 +0000 UTC m=+589.138299632" watchObservedRunningTime="2026-09-15 21:25:48.504229212 +0000 UTC m=+589.139381322" Sep 15 21:25:54.500657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:54.500623 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-72dr8" Sep 15 21:25:59.738294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:59.738262 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:25:59.738812 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:25:59.738794 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:26:40.378435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.378396 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cjf7f"] Sep 15 21:26:40.381806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.381789 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.384775 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.384754 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 15 21:26:40.385120 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.385099 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cqkvg\"" Sep 15 21:26:40.400435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.400403 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cjf7f"] Sep 15 21:26:40.445412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.445375 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zhz\" (UniqueName: \"kubernetes.io/projected/bff5fb03-322b-405e-8f20-e4767ba57873-kube-api-access-q7zhz\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.445574 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.445423 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff5fb03-322b-405e-8f20-e4767ba57873-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.546070 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.546037 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zhz\" (UniqueName: \"kubernetes.io/projected/bff5fb03-322b-405e-8f20-e4767ba57873-kube-api-access-q7zhz\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.546070 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.546073 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff5fb03-322b-405e-8f20-e4767ba57873-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.546798 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.546778 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff5fb03-322b-405e-8f20-e4767ba57873-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.562709 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.562686 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zhz\" (UniqueName: \"kubernetes.io/projected/bff5fb03-322b-405e-8f20-e4767ba57873-kube-api-access-q7zhz\") pod \"payload-processing-5b54b7fbb7-cjf7f\" (UID: \"bff5fb03-322b-405e-8f20-e4767ba57873\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.691404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.691329 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:40.992348 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.992274 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cjf7f"] Sep 15 21:26:40.995905 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:26:40.995875 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff5fb03_322b_405e_8f20_e4767ba57873.slice/crio-bad99f277536963a1aae88c10536e26b6ffaa3d909a43b68c7e6f9ec889cb2c5 WatchSource:0}: Error finding container bad99f277536963a1aae88c10536e26b6ffaa3d909a43b68c7e6f9ec889cb2c5: Status 404 returned error can't find the container with id bad99f277536963a1aae88c10536e26b6ffaa3d909a43b68c7e6f9ec889cb2c5 Sep 15 21:26:40.997985 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:40.997962 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:26:41.747508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:41.747473 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" event={"ID":"bff5fb03-322b-405e-8f20-e4767ba57873","Type":"ContainerStarted","Data":"bad99f277536963a1aae88c10536e26b6ffaa3d909a43b68c7e6f9ec889cb2c5"} Sep 15 21:26:42.754204 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:42.754155 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" event={"ID":"bff5fb03-322b-405e-8f20-e4767ba57873","Type":"ContainerStarted","Data":"d6fbc2f5e4f6cdbf6eac588cd967305801cf4474bf42ff73dfa2d5863f87de66"} Sep 15 21:26:42.754627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:42.754367 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:26:42.772422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:42.772375 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" podStartSLOduration=1.5674489870000001 podStartE2EDuration="2.772360311s" totalImagesPullingTime="1.204911324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:42.769738539 +0000 UTC m=+643.404890640" watchObservedRunningTime="2026-09-15 21:26:42.772360311 +0000 UTC m=+643.407512444" Sep 15 21:26:53.761550 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:26:53.761519 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cjf7f" Sep 15 21:27:10.214097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.214055 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd"] Sep 15 21:27:10.217795 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.217775 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.219936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.219907 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fxwc6\"" Sep 15 21:27:10.219936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.219907 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 21:27:10.220421 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.220397 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 15 21:27:10.220519 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.220400 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 21:27:10.228180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.228157 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd"] Sep 15 21:27:10.382884 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.382851 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.383045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.382907 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.383045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.382973 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.383045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.382998 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67585424-e2c1-471a-b126-823e0fd354bc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.383210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.383071 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.383210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.383100 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29tv\" (UniqueName: \"kubernetes.io/projected/67585424-e2c1-471a-b126-823e0fd354bc-kube-api-access-z29tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484063 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484003 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484063 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67585424-e2c1-471a-b126-823e0fd354bc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484084 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z29tv\" (UniqueName: \"kubernetes.io/projected/67585424-e2c1-471a-b126-823e0fd354bc-kube-api-access-z29tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484125 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484147 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484434 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484420 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484531 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484509 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.484588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.484539 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.486233 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.486205 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67585424-e2c1-471a-b126-823e0fd354bc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.486518 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.486498 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67585424-e2c1-471a-b126-823e0fd354bc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.491484 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.491463 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29tv\" (UniqueName: \"kubernetes.io/projected/67585424-e2c1-471a-b126-823e0fd354bc-kube-api-access-z29tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-trtsd\" (UID: \"67585424-e2c1-471a-b126-823e0fd354bc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.529092 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.529069 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:10.803725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.803701 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd"] Sep 15 21:27:10.805471 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:27:10.805435 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67585424_e2c1_471a_b126_823e0fd354bc.slice/crio-1db9434ca6c8bcd270b32f76d80ba78cd8261bc6b359210aaebe52e0e9844f5d WatchSource:0}: Error finding container 1db9434ca6c8bcd270b32f76d80ba78cd8261bc6b359210aaebe52e0e9844f5d: Status 404 returned error can't find the container with id 1db9434ca6c8bcd270b32f76d80ba78cd8261bc6b359210aaebe52e0e9844f5d Sep 15 21:27:10.892066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:10.892037 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" event={"ID":"67585424-e2c1-471a-b126-823e0fd354bc","Type":"ContainerStarted","Data":"1db9434ca6c8bcd270b32f76d80ba78cd8261bc6b359210aaebe52e0e9844f5d"} Sep 15 21:27:11.899137 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:11.899106 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" event={"ID":"67585424-e2c1-471a-b126-823e0fd354bc","Type":"ContainerStarted","Data":"086cd47e060e64e4f0db1870bd10c135ff6e77dd732ecd4602009306bf66c600"} Sep 15 21:27:11.917574 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:11.917529 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" podStartSLOduration=0.955535048 podStartE2EDuration="1.917513383s" totalImagesPullingTime="961.978335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:11.915720897 +0000 UTC m=+672.550873007" watchObservedRunningTime="2026-09-15 21:27:11.917513383 +0000 UTC m=+672.552665493" Sep 15 21:27:19.012882 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.012846 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h"] Sep 15 21:27:19.017489 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.017467 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.019516 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.019497 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 15 21:27:19.026160 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.026138 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h"] Sep 15 21:27:19.051641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051617 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww88c\" (UniqueName: \"kubernetes.io/projected/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-kube-api-access-ww88c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.051769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051682 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.051769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051747 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.051932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051789 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.051932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051821 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.051932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.051878 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.152806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.152780 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.152953 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.152813 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.152947 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.152980 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153123 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.153045 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww88c\" (UniqueName: \"kubernetes.io/projected/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-kube-api-access-ww88c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153123 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.153102 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.153203 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.153352 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.153661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.153392 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.155187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.155162 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.155275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.155258 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.161108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.161086 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww88c\" (UniqueName: \"kubernetes.io/projected/73ecfb64-b4e3-409f-a7d9-b725d9a0c793-kube-api-access-ww88c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tlc7h\" (UID: \"73ecfb64-b4e3-409f-a7d9-b725d9a0c793\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.328670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.328611 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:19.604262 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.602060 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h"] Sep 15 21:27:19.939460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.939423 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" event={"ID":"73ecfb64-b4e3-409f-a7d9-b725d9a0c793","Type":"ContainerStarted","Data":"f596fbda0ec17ddb4784b037fa1c972a20fc438f68c24c912b018f4198c4a02f"} Sep 15 21:27:19.939588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.939471 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" event={"ID":"73ecfb64-b4e3-409f-a7d9-b725d9a0c793","Type":"ContainerStarted","Data":"1126b7282c10fd3b915502daaada16f5964f48faa34c0565dee8866c0c647c45"} Sep 15 21:27:19.960490 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:19.960449 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" podStartSLOduration=1.724212533 podStartE2EDuration="1.960434265s" totalImagesPullingTime="236.221732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:19.957558348 +0000 UTC m=+680.592710459" watchObservedRunningTime="2026-09-15 21:27:19.960434265 +0000 UTC m=+680.595586376" Sep 15 21:27:20.529718 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:20.529675 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:20.530138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:20.529756 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:20.542079 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:20.542053 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:20.955705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:20.955680 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-trtsd" Sep 15 21:27:29.329774 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:29.329739 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:29.330237 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:29.329795 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:29.341874 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:29.341847 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:27:30.001159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:27:30.001130 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tlc7h" Sep 15 21:29:15.575084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.575052 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:29:15.578493 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.578461 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.580529 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.580499 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ac4411-openshift-default-dockercfg-d2xcc\"" Sep 15 21:29:15.592563 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.592541 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:29:15.636796 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.636772 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.636952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.636817 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.636952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.636881 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.636971 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.637016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.637086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.637132 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhgk4\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.637164 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.637310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.637189 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.737994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.737962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738006 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738036 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhgk4\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738088 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738235 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738291 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738384 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738389 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738430 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738456 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.738802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738779 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.739013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.738995 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.740486 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.740458 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.740586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.740544 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.746168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.746142 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.746307 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.746280 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhgk4\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4\") pod \"e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:15.889528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:15.889496 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:16.168334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:16.168261 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:29:16.171639 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:16.171608 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf73f6e_ba2b_4a0e_8b1f_8c21db8444db.slice/crio-d0c8c5c37ad3872f35c07d40c5e28e29820d782e1925d5888af33b3c117dbe2c WatchSource:0}: Error finding container d0c8c5c37ad3872f35c07d40c5e28e29820d782e1925d5888af33b3c117dbe2c: Status 404 returned error can't find the container with id d0c8c5c37ad3872f35c07d40c5e28e29820d782e1925d5888af33b3c117dbe2c Sep 15 21:29:16.532665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:16.532592 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerStarted","Data":"d0c8c5c37ad3872f35c07d40c5e28e29820d782e1925d5888af33b3c117dbe2c"} Sep 15 21:29:20.012108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.012068 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:20.012411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.012141 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:20.012411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.012183 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:20.560252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.560220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerStarted","Data":"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c"} Sep 15 21:29:20.588393 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.588342 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podStartSLOduration=1.750167585 podStartE2EDuration="5.588326722s" totalImagesPullingTime="3.838159137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:20.586505367 +0000 UTC m=+801.221657505" watchObservedRunningTime="2026-09-15 21:29:20.588326722 +0000 UTC m=+801.223478826" Sep 15 21:29:20.744589 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.744555 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:29:20.748186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.748168 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.750986 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.750965 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b99d7b-dockercfg-snttw\"" Sep 15 21:29:20.751505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.751488 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b99d7b\"" Sep 15 21:29:20.764573 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.764548 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:29:20.881391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.881368 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.881547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.881423 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5sg\" (UniqueName: \"kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.889847 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.889807 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:20.982316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.982291 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.982426 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.982343 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5sg\" (UniqueName: \"kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.983084 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.983067 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:20.996035 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:20.996011 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5sg\" (UniqueName: \"kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg\") pod \"payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:21.058650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.058625 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:21.340816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.340793 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:29:21.342362 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:21.342337 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea37b66_d37c_4fb5_a5d1_e4ce3f9e1ac7.slice/crio-ad8b2d2a4ba094add3e7bae4fce52f14934b8cb1a1fa73a6d5e54ebbd4184cd9 WatchSource:0}: Error finding container ad8b2d2a4ba094add3e7bae4fce52f14934b8cb1a1fa73a6d5e54ebbd4184cd9: Status 404 returned error can't find the container with id ad8b2d2a4ba094add3e7bae4fce52f14934b8cb1a1fa73a6d5e54ebbd4184cd9 Sep 15 21:29:21.568315 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.568279 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" event={"ID":"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7","Type":"ContainerStarted","Data":"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e"} Sep 15 21:29:21.568577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.568544 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" event={"ID":"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7","Type":"ContainerStarted","Data":"ad8b2d2a4ba094add3e7bae4fce52f14934b8cb1a1fa73a6d5e54ebbd4184cd9"} Sep 15 21:29:21.568670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.568585 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:21.894380 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.894352 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:21.986640 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:21.986589 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" podStartSLOduration=1.986573296 podStartE2EDuration="1.986573296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:21.619317313 +0000 UTC m=+802.254469427" watchObservedRunningTime="2026-09-15 21:29:21.986573296 +0000 UTC m=+802.621725460" Sep 15 21:29:22.312411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.312311 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:29:22.315821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.315802 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.317976 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.317958 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4bc420\"" Sep 15 21:29:22.329229 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.329208 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:29:22.395108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.395081 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.395218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.395136 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tt9s\" (UniqueName: \"kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.495771 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.495735 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.495942 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.495782 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tt9s\" (UniqueName: \"kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.496435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.496417 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.504659 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.504633 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tt9s\" (UniqueName: \"kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s\") pod \"payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.574616 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.574542 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:22.575197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.575178 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:22.628993 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.628724 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:22.916141 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:22.916115 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:29:22.918661 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:22.918628 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabf3e48_4951_48ac_ad52_b76cf422ea85.slice/crio-3d0beb2b9bbf4f30f08c8594b8c920ead62363179cc1844a791deb795d17810a WatchSource:0}: Error finding container 3d0beb2b9bbf4f30f08c8594b8c920ead62363179cc1844a791deb795d17810a: Status 404 returned error can't find the container with id 3d0beb2b9bbf4f30f08c8594b8c920ead62363179cc1844a791deb795d17810a Sep 15 21:29:23.581883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:23.581838 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" event={"ID":"dabf3e48-4951-48ac-ad52-b76cf422ea85","Type":"ContainerStarted","Data":"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af"} Sep 15 21:29:23.582343 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:23.581895 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" event={"ID":"dabf3e48-4951-48ac-ad52-b76cf422ea85","Type":"ContainerStarted","Data":"3d0beb2b9bbf4f30f08c8594b8c920ead62363179cc1844a791deb795d17810a"} Sep 15 21:29:23.582343 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:23.581996 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:23.598290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:23.598248 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" podStartSLOduration=1.5982327349999998 podStartE2EDuration="1.598232735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:23.597116442 +0000 UTC m=+804.232268550" watchObservedRunningTime="2026-09-15 21:29:23.598232735 +0000 UTC m=+804.233384846" Sep 15 21:29:26.528844 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.528802 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:29:26.532215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.532196 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.535530 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.535510 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ac4411\"" Sep 15 21:29:26.535651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.535635 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ac4411-dockercfg-tn9fk\"" Sep 15 21:29:26.544962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.544941 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:29:26.548232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.548213 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:29:26.548306 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.548290 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.560957 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.560936 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:29:26.632294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.631943 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kkz\" (UniqueName: \"kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.632294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.632045 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.632294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.632075 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.632294 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.632098 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhfj\" (UniqueName: \"kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.732957 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.732915 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kkz\" (UniqueName: \"kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.733142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.733009 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.733142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.733039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.733142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.733064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhfj\" (UniqueName: \"kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.733714 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.733693 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.733766 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.733700 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.742215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.742194 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhfj\" (UniqueName: \"kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj\") pod \"payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:26.742327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.742197 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kkz\" (UniqueName: \"kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz\") pod \"payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.842981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.842916 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:26.858588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:26.858557 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:27.129741 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.129718 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:29:27.130588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:27.130566 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2da6711_1939_415c_9ae6_4536dbeed30d.slice/crio-f7da071baa645eb6154b7f628a93905ef282b34c05debddd44b7afa7093c2d8e WatchSource:0}: Error finding container f7da071baa645eb6154b7f628a93905ef282b34c05debddd44b7afa7093c2d8e: Status 404 returned error can't find the container with id f7da071baa645eb6154b7f628a93905ef282b34c05debddd44b7afa7093c2d8e Sep 15 21:29:27.157076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.157055 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:29:27.162852 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:27.162801 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8846e0f9_cc3e_4c43_86ec_7eb27ef4df23.slice/crio-a8b0d1929901f456f71659c41d264a1d19cdc818597e85496b9fb5916680893a WatchSource:0}: Error finding container a8b0d1929901f456f71659c41d264a1d19cdc818597e85496b9fb5916680893a: Status 404 returned error can't find the container with id a8b0d1929901f456f71659c41d264a1d19cdc818597e85496b9fb5916680893a Sep 15 21:29:27.604290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.604194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" event={"ID":"e2da6711-1939-415c-9ae6-4536dbeed30d","Type":"ContainerStarted","Data":"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22"} Sep 15 21:29:27.604290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.604249 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" event={"ID":"e2da6711-1939-415c-9ae6-4536dbeed30d","Type":"ContainerStarted","Data":"f7da071baa645eb6154b7f628a93905ef282b34c05debddd44b7afa7093c2d8e"} Sep 15 21:29:27.604743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.604300 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:27.605773 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.605747 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" event={"ID":"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23","Type":"ContainerStarted","Data":"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277"} Sep 15 21:29:27.605895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.605795 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:27.605895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.605812 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" event={"ID":"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23","Type":"ContainerStarted","Data":"a8b0d1929901f456f71659c41d264a1d19cdc818597e85496b9fb5916680893a"} Sep 15 21:29:27.645805 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.645752 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" podStartSLOduration=1.645734884 podStartE2EDuration="1.645734884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:27.637403843 +0000 UTC m=+808.272555966" watchObservedRunningTime="2026-09-15 21:29:27.645734884 +0000 UTC m=+808.280886996" Sep 15 21:29:27.689153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:27.689110 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" podStartSLOduration=1.689096734 podStartE2EDuration="1.689096734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:27.686257198 +0000 UTC m=+808.321409309" watchObservedRunningTime="2026-09-15 21:29:27.689096734 +0000 UTC m=+808.324248845" Sep 15 21:29:32.195696 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.195660 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:32.202330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.202307 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.205291 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.205265 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c3e1d9c1\"" Sep 15 21:29:32.205399 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.205370 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c3e1d9c1-dockercfg-5vtwc\"" Sep 15 21:29:32.213857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.213816 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:32.235645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.235615 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:32.239718 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.239700 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.257677 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.257655 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:32.383397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.383364 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljz5\" (UniqueName: \"kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.383397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.383401 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.383639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.383425 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.383639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.383480 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ww4\" (UniqueName: \"kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.484232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.484168 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ww4\" (UniqueName: \"kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.484367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.484270 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljz5\" (UniqueName: \"kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.484367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.484311 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.484367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.484356 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.485078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.485055 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.485159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.485062 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.492663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.492644 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ww4\" (UniqueName: \"kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4\") pod \"payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.492863 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.492821 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljz5\" (UniqueName: \"kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5\") pod \"payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.513592 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.513570 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:32.550055 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.550029 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:32.578374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.577399 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:29:32.800690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.800664 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:32.802799 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:32.802762 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d84a13d_b3ee_426f_8ba0_b2b152081f8a.slice/crio-615c3023f782317dfc6897a63ac4da6dd35df5b37382e74f9f015ed36a300abb WatchSource:0}: Error finding container 615c3023f782317dfc6897a63ac4da6dd35df5b37382e74f9f015ed36a300abb: Status 404 returned error can't find the container with id 615c3023f782317dfc6897a63ac4da6dd35df5b37382e74f9f015ed36a300abb Sep 15 21:29:32.844498 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:32.844469 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:32.845768 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:32.845741 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b98f12_fc69_4bc1_b5a9_9c51b80d5583.slice/crio-1a950b320b1fed27db17db096a0fad178d72799b6d7b85d75450e0e1e9b0326a WatchSource:0}: Error finding container 1a950b320b1fed27db17db096a0fad178d72799b6d7b85d75450e0e1e9b0326a: Status 404 returned error can't find the container with id 1a950b320b1fed27db17db096a0fad178d72799b6d7b85d75450e0e1e9b0326a Sep 15 21:29:33.542483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.542448 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:29:33.546172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.546155 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.548445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.548421 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-10c104-dockercfg-gnkwf\"" Sep 15 21:29:33.548569 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.548424 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-10c104\"" Sep 15 21:29:33.562811 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.562788 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:29:33.650670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.650625 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" event={"ID":"9d84a13d-b3ee-426f-8ba0-b2b152081f8a","Type":"ContainerStarted","Data":"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e"} Sep 15 21:29:33.650857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.650701 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:33.650857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.650725 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" event={"ID":"9d84a13d-b3ee-426f-8ba0-b2b152081f8a","Type":"ContainerStarted","Data":"615c3023f782317dfc6897a63ac4da6dd35df5b37382e74f9f015ed36a300abb"} Sep 15 21:29:33.652266 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.652245 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" event={"ID":"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583","Type":"ContainerStarted","Data":"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814"} Sep 15 21:29:33.652381 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.652292 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" event={"ID":"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583","Type":"ContainerStarted","Data":"1a950b320b1fed27db17db096a0fad178d72799b6d7b85d75450e0e1e9b0326a"} Sep 15 21:29:33.652419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.652386 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:33.675981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.675928 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" podStartSLOduration=1.675913886 podStartE2EDuration="1.675913886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:33.675326882 +0000 UTC m=+814.310478993" watchObservedRunningTime="2026-09-15 21:29:33.675913886 +0000 UTC m=+814.311065996" Sep 15 21:29:33.695859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.695821 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.695982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.695885 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5cd\" (UniqueName: \"kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.696407 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.696373 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" podStartSLOduration=1.696359961 podStartE2EDuration="1.696359961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:33.694563435 +0000 UTC m=+814.329715545" watchObservedRunningTime="2026-09-15 21:29:33.696359961 +0000 UTC m=+814.331512122" Sep 15 21:29:33.797400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.797335 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.797400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.797383 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5cd\" (UniqueName: \"kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.798195 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.798176 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.808450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.808424 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5cd\" (UniqueName: \"kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd\") pod \"payload-pre-processing-e2e-autores-10c104-888747cbb-258k9\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:33.857353 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:33.857323 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:34.141750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.141711 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:29:34.145469 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:34.145438 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f55973_3f5a_4d37_933d_cb509168fbf8.slice/crio-081a5b67575c7c7131de3be72e9bfb3a93d1e5e18f070eca6f470369fc8ca615 WatchSource:0}: Error finding container 081a5b67575c7c7131de3be72e9bfb3a93d1e5e18f070eca6f470369fc8ca615: Status 404 returned error can't find the container with id 081a5b67575c7c7131de3be72e9bfb3a93d1e5e18f070eca6f470369fc8ca615 Sep 15 21:29:34.589556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.589486 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:29:34.660612 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.660548 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" event={"ID":"f4f55973-3f5a-4d37-933d-cb509168fbf8","Type":"ContainerStarted","Data":"2a3839a0c37da318c7ebc6c68bca9222637e69148850c10de3c0684ecd18114a"} Sep 15 21:29:34.660876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.660856 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:34.660983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.660890 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" event={"ID":"f4f55973-3f5a-4d37-933d-cb509168fbf8","Type":"ContainerStarted","Data":"081a5b67575c7c7131de3be72e9bfb3a93d1e5e18f070eca6f470369fc8ca615"} Sep 15 21:29:34.681631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:34.681589 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" podStartSLOduration=1.681573347 podStartE2EDuration="1.681573347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:34.680790286 +0000 UTC m=+815.315942397" watchObservedRunningTime="2026-09-15 21:29:34.681573347 +0000 UTC m=+815.316725458" Sep 15 21:29:38.619877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:38.619822 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:29:38.620225 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:38.619926 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:29:44.664394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:44.664359 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:44.664886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:44.664449 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:45.669014 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:45.668983 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:29:46.313503 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.313466 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:29:46.320617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.320600 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.322808 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.322789 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bb7867\"" Sep 15 21:29:46.322901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.322885 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bb7867-dockercfg-wvj2r\"" Sep 15 21:29:46.332238 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.332215 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:29:46.399473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.399431 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.399621 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.399505 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lw69\" (UniqueName: \"kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.500352 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.500323 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.500448 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.500366 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lw69\" (UniqueName: \"kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.501092 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.501071 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.509002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.508974 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lw69\" (UniqueName: \"kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69\") pod \"payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.632060 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.632023 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:46.915523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:46.915491 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:29:46.916072 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:46.916044 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575bcf76_84b1_446d_9282_a71bc56ee3dc.slice/crio-addad44678b7dd1ee5f41e31c758671452e37230f256573d1c93dcd20b401638 WatchSource:0}: Error finding container addad44678b7dd1ee5f41e31c758671452e37230f256573d1c93dcd20b401638: Status 404 returned error can't find the container with id addad44678b7dd1ee5f41e31c758671452e37230f256573d1c93dcd20b401638 Sep 15 21:29:47.445309 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.445270 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8"] Sep 15 21:29:47.449688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.449660 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:47.452274 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.452064 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:29:47.454698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.454671 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8"] Sep 15 21:29:47.509994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.509954 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8\" (UID: \"91483cff-6b81-4b64-aad6-c273baefa8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:47.611300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.611257 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8\" (UID: \"91483cff-6b81-4b64-aad6-c273baefa8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:47.612093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.612067 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8\" (UID: \"91483cff-6b81-4b64-aad6-c273baefa8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:47.744122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.744038 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" event={"ID":"575bcf76-84b1-446d-9282-a71bc56ee3dc","Type":"ContainerStarted","Data":"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c"} Sep 15 21:29:47.744122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.744096 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" event={"ID":"575bcf76-84b1-446d-9282-a71bc56ee3dc","Type":"ContainerStarted","Data":"addad44678b7dd1ee5f41e31c758671452e37230f256573d1c93dcd20b401638"} Sep 15 21:29:47.744350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.744335 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:47.764617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.764564 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" podStartSLOduration=1.764547817 podStartE2EDuration="1.764547817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:47.761608855 +0000 UTC m=+828.396760965" watchObservedRunningTime="2026-09-15 21:29:47.764547817 +0000 UTC m=+828.399699952" Sep 15 21:29:47.765043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:47.765015 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:48.050341 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:48.050294 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8"] Sep 15 21:29:48.050754 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:29:48.050471 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91483cff_6b81_4b64_aad6_c273baefa8e7.slice/crio-3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925 WatchSource:0}: Error finding container 3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925: Status 404 returned error can't find the container with id 3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925 Sep 15 21:29:48.752552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:48.752515 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" event={"ID":"91483cff-6b81-4b64-aad6-c273baefa8e7","Type":"ContainerStarted","Data":"3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925"} Sep 15 21:29:50.769305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.769194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" event={"ID":"91483cff-6b81-4b64-aad6-c273baefa8e7","Type":"ContainerStarted","Data":"904ccc8bc97965232bd28c58249cc746c5ad2f80171797f23eead70887bd80a9"} Sep 15 21:29:50.771439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.771416 2552 generic.go:356] "Generic (PLEG): container finished" podID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerID="35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c" exitCode=0 Sep 15 21:29:50.771582 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.771490 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerDied","Data":"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c"} Sep 15 21:29:50.771854 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.771815 2552 scope.go:117] "RemoveContainer" containerID="35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c" Sep 15 21:29:50.772943 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.772919 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:50.773015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.772977 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:50.773015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.773005 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:29:50.785901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:50.785855 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" podStartSLOduration=1.6197404899999999 podStartE2EDuration="3.785823338s" totalImagesPullingTime="2.166082848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:50.784320648 +0000 UTC m=+831.419472759" watchObservedRunningTime="2026-09-15 21:29:50.785823338 +0000 UTC m=+831.420975449" Sep 15 21:29:51.778529 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.778493 2552 generic.go:356] "Generic (PLEG): container finished" podID="91483cff-6b81-4b64-aad6-c273baefa8e7" containerID="904ccc8bc97965232bd28c58249cc746c5ad2f80171797f23eead70887bd80a9" exitCode=0 Sep 15 21:29:51.778944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.778586 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" event={"ID":"91483cff-6b81-4b64-aad6-c273baefa8e7","Type":"ContainerDied","Data":"904ccc8bc97965232bd28c58249cc746c5ad2f80171797f23eead70887bd80a9"} Sep 15 21:29:51.780584 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.780553 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerStarted","Data":"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd"} Sep 15 21:29:51.890379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.890347 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:51.891541 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.891517 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:51.891654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:51.891559 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:52.575312 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.575271 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:52.890291 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.890258 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:52.890722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.890334 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:52.937926 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.937904 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:52.966718 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.966677 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca\") pod \"91483cff-6b81-4b64-aad6-c273baefa8e7\" (UID: \"91483cff-6b81-4b64-aad6-c273baefa8e7\") " Sep 15 21:29:52.967759 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:52.967281 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91483cff-6b81-4b64-aad6-c273baefa8e7" (UID: "91483cff-6b81-4b64-aad6-c273baefa8e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:29:53.068459 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.068421 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91483cff-6b81-4b64-aad6-c273baefa8e7-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:29:53.793372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.793341 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" Sep 15 21:29:53.793567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.793385 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8" event={"ID":"91483cff-6b81-4b64-aad6-c273baefa8e7","Type":"ContainerDied","Data":"3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925"} Sep 15 21:29:53.793567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.793427 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8cf3da075c55932bf2df141974a60da33942619ebda227df6ae47cbe74f925" Sep 15 21:29:53.889959 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.889928 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:53.890125 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:53.889984 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:54.890150 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:54.890117 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:54.890508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:54.890168 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:55.890450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:55.890412 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:55.890946 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:55.890474 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:56.890737 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:56.890703 2552 patch_prober.go:28] interesting pod/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 15 21:29:56.891133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:56.890752 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 15 21:29:57.894268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:57.894236 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:57.895358 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:57.895336 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:29:58.121664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.121634 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:58.121895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.121867 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" podUID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" containerName="payload-processing" containerID="cri-o://3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814" gracePeriod=30 Sep 15 21:29:58.133273 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.133238 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:58.133505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.133476 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" podUID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" containerName="payload-pre-processing" containerID="cri-o://6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e" gracePeriod=30 Sep 15 21:29:58.324300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.324265 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8"] Sep 15 21:29:58.331727 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.331697 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e1d9c1-bkgw8"] Sep 15 21:29:58.409310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.409286 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:58.412088 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.412070 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:58.515339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515263 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28ww4\" (UniqueName: \"kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4\") pod \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " Sep 15 21:29:58.515339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515299 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume\") pod \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " Sep 15 21:29:58.515339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515337 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume\") pod \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\" (UID: \"9d84a13d-b3ee-426f-8ba0-b2b152081f8a\") " Sep 15 21:29:58.515605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515409 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pljz5\" (UniqueName: \"kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5\") pod \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\" (UID: \"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583\") " Sep 15 21:29:58.515724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515699 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" (UID: "a2b98f12-fc69-4bc1-b5a9-9c51b80d5583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:29:58.515777 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.515717 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d84a13d-b3ee-426f-8ba0-b2b152081f8a" (UID: "9d84a13d-b3ee-426f-8ba0-b2b152081f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:29:58.517326 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.517303 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5" (OuterVolumeSpecName: "kube-api-access-pljz5") pod "a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" (UID: "a2b98f12-fc69-4bc1-b5a9-9c51b80d5583"). InnerVolumeSpecName "kube-api-access-pljz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:29:58.517465 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.517444 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4" (OuterVolumeSpecName: "kube-api-access-28ww4") pod "9d84a13d-b3ee-426f-8ba0-b2b152081f8a" (UID: "9d84a13d-b3ee-426f-8ba0-b2b152081f8a"). InnerVolumeSpecName "kube-api-access-28ww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:29:58.617133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.617071 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:29:58.617133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.617129 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljz5\" (UniqueName: \"kubernetes.io/projected/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-kube-api-access-pljz5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:29:58.617133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.617142 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ww4\" (UniqueName: \"kubernetes.io/projected/9d84a13d-b3ee-426f-8ba0-b2b152081f8a-kube-api-access-28ww4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:29:58.617358 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.617151 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:29:58.754761 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.754732 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:29:58.823382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.823288 2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" containerID="3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814" exitCode=0 Sep 15 21:29:58.823382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.823322 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" Sep 15 21:29:58.823382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.823371 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" event={"ID":"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583","Type":"ContainerDied","Data":"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814"} Sep 15 21:29:58.823659 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.823419 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph" event={"ID":"a2b98f12-fc69-4bc1-b5a9-9c51b80d5583","Type":"ContainerDied","Data":"1a950b320b1fed27db17db096a0fad178d72799b6d7b85d75450e0e1e9b0326a"} Sep 15 21:29:58.823659 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.823440 2552 scope.go:117] "RemoveContainer" containerID="3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814" Sep 15 21:29:58.825037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.825014 2552 generic.go:356] "Generic (PLEG): container finished" podID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" containerID="6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e" exitCode=0 Sep 15 21:29:58.825153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.825045 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" event={"ID":"9d84a13d-b3ee-426f-8ba0-b2b152081f8a","Type":"ContainerDied","Data":"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e"} Sep 15 21:29:58.825153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.825069 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" Sep 15 21:29:58.825153 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.825097 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5" event={"ID":"9d84a13d-b3ee-426f-8ba0-b2b152081f8a","Type":"ContainerDied","Data":"615c3023f782317dfc6897a63ac4da6dd35df5b37382e74f9f015ed36a300abb"} Sep 15 21:29:58.833668 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.833643 2552 scope.go:117] "RemoveContainer" containerID="3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814" Sep 15 21:29:58.834080 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:29:58.834045 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814\": container with ID starting with 3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814 not found: ID does not exist" containerID="3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814" Sep 15 21:29:58.834164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.834084 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814"} err="failed to get container status \"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814\": rpc error: code = NotFound desc = could not find container \"3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814\": container with ID starting with 3a06434ff68869ac673018f37f9ee9aa17d3283ce9f30cdfc4f2fa87c548f814 not found: ID does not exist" Sep 15 21:29:58.834164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.834106 2552 scope.go:117] "RemoveContainer" containerID="6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e" Sep 15 21:29:58.842527 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.842511 2552 scope.go:117] "RemoveContainer" containerID="6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e" Sep 15 21:29:58.842763 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:29:58.842743 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e\": container with ID starting with 6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e not found: ID does not exist" containerID="6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e" Sep 15 21:29:58.842857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.842771 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e"} err="failed to get container status \"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e\": rpc error: code = NotFound desc = could not find container \"6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e\": container with ID starting with 6ea897e03adec0cb3056cccdc706cbf553ebf6dad368f2059b632211a1b13a2e not found: ID does not exist" Sep 15 21:29:58.855187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.855161 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:58.857736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.857718 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e1d9c1-756f664dfd-5bbb5"] Sep 15 21:29:58.866259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.866240 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:58.879444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:58.879418 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e1d9c1-7445f7fb4f-flcph"] Sep 15 21:29:59.791850 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:59.791791 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91483cff-6b81-4b64-aad6-c273baefa8e7" path="/var/lib/kubelet/pods/91483cff-6b81-4b64-aad6-c273baefa8e7/volumes" Sep 15 21:29:59.792218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:59.792186 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" path="/var/lib/kubelet/pods/9d84a13d-b3ee-426f-8ba0-b2b152081f8a/volumes" Sep 15 21:29:59.792504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:29:59.792489 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" path="/var/lib/kubelet/pods/a2b98f12-fc69-4bc1-b5a9-9c51b80d5583/volumes" Sep 15 21:30:00.191069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191031 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg"] Sep 15 21:30:00.191430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191417 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91483cff-6b81-4b64-aad6-c273baefa8e7" containerName="revoke-keys" Sep 15 21:30:00.191469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191432 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="91483cff-6b81-4b64-aad6-c273baefa8e7" containerName="revoke-keys" Sep 15 21:30:00.191469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191444 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" containerName="payload-pre-processing" Sep 15 21:30:00.191469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191450 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" containerName="payload-pre-processing" Sep 15 21:30:00.191469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191465 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" containerName="payload-processing" Sep 15 21:30:00.191609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191471 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" containerName="payload-processing" Sep 15 21:30:00.191609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191536 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b98f12-fc69-4bc1-b5a9-9c51b80d5583" containerName="payload-processing" Sep 15 21:30:00.191609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191546 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d84a13d-b3ee-426f-8ba0-b2b152081f8a" containerName="payload-pre-processing" Sep 15 21:30:00.191609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.191552 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="91483cff-6b81-4b64-aad6-c273baefa8e7" containerName="revoke-keys" Sep 15 21:30:00.196300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.196278 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" Sep 15 21:30:00.198029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.198002 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln"] Sep 15 21:30:00.201429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.201411 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz"] Sep 15 21:30:00.201526 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.201508 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" Sep 15 21:30:00.204999 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.204979 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" Sep 15 21:30:00.216598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.216582 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:30:00.217646 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.217592 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" Sep 15 21:30:00.220430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.220403 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln"] Sep 15 21:30:00.222815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.222793 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" Sep 15 21:30:00.224071 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.223993 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg"] Sep 15 21:30:00.225310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.225291 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" Sep 15 21:30:00.227858 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.227821 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz"] Sep 15 21:30:00.304268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.304235 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9"] Sep 15 21:30:00.310676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.310000 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" Sep 15 21:30:00.322750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.322720 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9"] Sep 15 21:30:00.331790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.331620 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" Sep 15 21:30:00.507861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.507837 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg"] Sep 15 21:30:00.509511 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:00.509486 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9df8ace_7812_49b3_b40c_fc46cb10ed82.slice/crio-7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82 WatchSource:0}: Error finding container 7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82: Status 404 returned error can't find the container with id 7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82 Sep 15 21:30:00.534268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.534242 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln"] Sep 15 21:30:00.537533 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.537513 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz"] Sep 15 21:30:00.541355 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:00.541329 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf7cdca_e0e9_4803_826a_69f3e62f4e17.slice/crio-2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487 WatchSource:0}: Error finding container 2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487: Status 404 returned error can't find the container with id 2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487 Sep 15 21:30:00.541547 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:00.541525 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f9d0ff_24c1_48bf_9b71_1ec0b1da9bb5.slice/crio-1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05 WatchSource:0}: Error finding container 1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05: Status 404 returned error can't find the container with id 1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05 Sep 15 21:30:00.634152 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.634127 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9"] Sep 15 21:30:00.635512 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:00.635488 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e64bf_9f9e_44bd_a41b_42b5286e1117.slice/crio-fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472 WatchSource:0}: Error finding container fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472: Status 404 returned error can't find the container with id fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472 Sep 15 21:30:00.840210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.840170 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" event={"ID":"54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5","Type":"ContainerStarted","Data":"1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05"} Sep 15 21:30:00.841851 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.841793 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" event={"ID":"bdf7cdca-e0e9-4803-826a-69f3e62f4e17","Type":"ContainerStarted","Data":"2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487"} Sep 15 21:30:00.843460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.843430 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" event={"ID":"c70e64bf-9f9e-44bd-a41b-42b5286e1117","Type":"ContainerStarted","Data":"fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472"} Sep 15 21:30:00.845155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:00.845130 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" event={"ID":"c9df8ace-7812-49b3-b40c-fc46cb10ed82","Type":"ContainerStarted","Data":"7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82"} Sep 15 21:30:01.852665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.852584 2552 generic.go:356] "Generic (PLEG): container finished" podID="c9df8ace-7812-49b3-b40c-fc46cb10ed82" containerID="b6b32130d91968a972fb67b42b39e12b2cfc26ad3179c4b33e23996f085714c7" exitCode=0 Sep 15 21:30:01.853082 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.852663 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" event={"ID":"c9df8ace-7812-49b3-b40c-fc46cb10ed82","Type":"ContainerDied","Data":"b6b32130d91968a972fb67b42b39e12b2cfc26ad3179c4b33e23996f085714c7"} Sep 15 21:30:01.854175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.854155 2552 generic.go:356] "Generic (PLEG): container finished" podID="54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5" containerID="0981798aefcbd934ab094dfb228a7fa85090e9150b3852af0f2512fd1f31aa8e" exitCode=0 Sep 15 21:30:01.854277 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.854221 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" event={"ID":"54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5","Type":"ContainerDied","Data":"0981798aefcbd934ab094dfb228a7fa85090e9150b3852af0f2512fd1f31aa8e"} Sep 15 21:30:01.855591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.855571 2552 generic.go:356] "Generic (PLEG): container finished" podID="bdf7cdca-e0e9-4803-826a-69f3e62f4e17" containerID="4f6f64c6d83ba5e5adc7ebe3d3eb566b376c555c87296f0c7e312e1493f5b236" exitCode=0 Sep 15 21:30:01.855673 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.855654 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" event={"ID":"bdf7cdca-e0e9-4803-826a-69f3e62f4e17","Type":"ContainerDied","Data":"4f6f64c6d83ba5e5adc7ebe3d3eb566b376c555c87296f0c7e312e1493f5b236"} Sep 15 21:30:01.856972 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.856954 2552 generic.go:356] "Generic (PLEG): container finished" podID="c70e64bf-9f9e-44bd-a41b-42b5286e1117" containerID="ee52a6e10e7a7b9add38220010ce5da8b2da15730bd8469e7405b61b02ff6b7e" exitCode=0 Sep 15 21:30:01.857049 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:01.856974 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" event={"ID":"c70e64bf-9f9e-44bd-a41b-42b5286e1117","Type":"ContainerDied","Data":"ee52a6e10e7a7b9add38220010ce5da8b2da15730bd8469e7405b61b02ff6b7e"} Sep 15 21:30:03.010094 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.010070 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" Sep 15 21:30:03.042461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.042436 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" Sep 15 21:30:03.085156 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.085138 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" Sep 15 21:30:03.091539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.091520 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" Sep 15 21:30:03.869061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.869039 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" Sep 15 21:30:03.869185 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.869065 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9" event={"ID":"c70e64bf-9f9e-44bd-a41b-42b5286e1117","Type":"ContainerDied","Data":"fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472"} Sep 15 21:30:03.869185 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.869107 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5e7175270b4e86fe2f8679eb0cbb407d15edda1b6188509b2df9c23c914472" Sep 15 21:30:03.870342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.870317 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" Sep 15 21:30:03.870447 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.870371 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg" event={"ID":"c9df8ace-7812-49b3-b40c-fc46cb10ed82","Type":"ContainerDied","Data":"7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82"} Sep 15 21:30:03.870447 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.870419 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eb14d52133ff5498543aaadfcb2d8ee010d90e2de7fa996216ba9cd64ca1c82" Sep 15 21:30:03.871999 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.871984 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" Sep 15 21:30:03.872095 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.872022 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz" event={"ID":"54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5","Type":"ContainerDied","Data":"1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05"} Sep 15 21:30:03.872095 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.872061 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf4a709eb8a866149bfad63eb3057f46c6cebaf23a2f1b692990f8f9c970f05" Sep 15 21:30:03.873731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.873700 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" event={"ID":"bdf7cdca-e0e9-4803-826a-69f3e62f4e17","Type":"ContainerDied","Data":"2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487"} Sep 15 21:30:03.873731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.873719 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln" Sep 15 21:30:03.873901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:03.873739 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2341bcf7b123d521abc2dd94991d54577aad777e55eed80f6fa36fdc6b4d9487" Sep 15 21:30:05.121641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.121608 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:05.122048 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122034 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9df8ace-7812-49b3-b40c-fc46cb10ed82" containerName="cleanup" Sep 15 21:30:05.122048 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122047 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9df8ace-7812-49b3-b40c-fc46cb10ed82" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122058 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122064 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122084 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70e64bf-9f9e-44bd-a41b-42b5286e1117" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122090 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70e64bf-9f9e-44bd-a41b-42b5286e1117" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122104 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf7cdca-e0e9-4803-826a-69f3e62f4e17" containerName="cleanup" Sep 15 21:30:05.122116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122109 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf7cdca-e0e9-4803-826a-69f3e62f4e17" containerName="cleanup" Sep 15 21:30:05.122286 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122172 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9df8ace-7812-49b3-b40c-fc46cb10ed82" containerName="cleanup" Sep 15 21:30:05.122286 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122181 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf7cdca-e0e9-4803-826a-69f3e62f4e17" containerName="cleanup" Sep 15 21:30:05.122286 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122187 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70e64bf-9f9e-44bd-a41b-42b5286e1117" containerName="cleanup" Sep 15 21:30:05.122286 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.122193 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5" containerName="cleanup" Sep 15 21:30:05.125379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.125360 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.128616 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.128593 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5a03be06\"" Sep 15 21:30:05.141697 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.141670 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:05.144321 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.144299 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:05.147734 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.147718 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.150284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.150266 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5a03be06-dockercfg-mhks6\"" Sep 15 21:30:05.160560 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.160537 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:05.182345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.182316 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.182480 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.182376 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rxc\" (UniqueName: \"kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.182480 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.182441 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.182480 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.182465 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwfs\" (UniqueName: \"kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.283251 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.283217 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rxc\" (UniqueName: \"kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.283396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.283259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.283396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.283277 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwfs\" (UniqueName: \"kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.283396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.283339 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.284070 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.284050 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.284200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.284056 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.295822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.295793 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rxc\" (UniqueName: \"kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc\") pod \"payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.304751 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.304730 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwfs\" (UniqueName: \"kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs\") pod \"payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.435951 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.435891 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.459099 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.459077 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.719279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.719248 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:05.719417 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:05.719310 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2d99af_ff58_4a94_8247_76675b37eeac.slice/crio-c98848c0824fe83c529480707fb6717a4a9658876dc9a6b8c8503f92a4c72b7d WatchSource:0}: Error finding container c98848c0824fe83c529480707fb6717a4a9658876dc9a6b8c8503f92a4c72b7d: Status 404 returned error can't find the container with id c98848c0824fe83c529480707fb6717a4a9658876dc9a6b8c8503f92a4c72b7d Sep 15 21:30:05.739687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.739660 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:05.740588 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:05.740552 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc686c71_1158_4d62_a0c7_9cf501aeca7d.slice/crio-9f5ee3aa16433df03ca61a32f7235f97018835954c65c8b9cdf23588e72faf98 WatchSource:0}: Error finding container 9f5ee3aa16433df03ca61a32f7235f97018835954c65c8b9cdf23588e72faf98: Status 404 returned error can't find the container with id 9f5ee3aa16433df03ca61a32f7235f97018835954c65c8b9cdf23588e72faf98 Sep 15 21:30:05.890439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.890333 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" event={"ID":"bc686c71-1158-4d62-a0c7-9cf501aeca7d","Type":"ContainerStarted","Data":"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f"} Sep 15 21:30:05.890439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.890399 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:05.890439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.890428 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" event={"ID":"bc686c71-1158-4d62-a0c7-9cf501aeca7d","Type":"ContainerStarted","Data":"9f5ee3aa16433df03ca61a32f7235f97018835954c65c8b9cdf23588e72faf98"} Sep 15 21:30:05.892026 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.891981 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" event={"ID":"bb2d99af-ff58-4a94-8247-76675b37eeac","Type":"ContainerStarted","Data":"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a"} Sep 15 21:30:05.892136 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.892048 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" event={"ID":"bb2d99af-ff58-4a94-8247-76675b37eeac","Type":"ContainerStarted","Data":"c98848c0824fe83c529480707fb6717a4a9658876dc9a6b8c8503f92a4c72b7d"} Sep 15 21:30:05.892136 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.892078 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:05.909848 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.909788 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" podStartSLOduration=0.909773967 podStartE2EDuration="909.773967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:05.907577511 +0000 UTC m=+846.542729622" watchObservedRunningTime="2026-09-15 21:30:05.909773967 +0000 UTC m=+846.544926078" Sep 15 21:30:05.927957 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:05.927911 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" podStartSLOduration=0.927897298 podStartE2EDuration="927.897298ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:05.925699635 +0000 UTC m=+846.560851727" watchObservedRunningTime="2026-09-15 21:30:05.927897298 +0000 UTC m=+846.563049408" Sep 15 21:30:16.899412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:16.899379 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:16.899954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:16.899554 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:27.283775 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.283741 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm"] Sep 15 21:30:27.287284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.287260 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:27.289353 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.289335 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:30:27.294678 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.294656 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm"] Sep 15 21:30:27.378987 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.378958 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm\" (UID: \"507a859a-3273-48d9-9f34-458c24d8e34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:27.479605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.479577 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm\" (UID: \"507a859a-3273-48d9-9f34-458c24d8e34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:27.480296 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.480268 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm\" (UID: \"507a859a-3273-48d9-9f34-458c24d8e34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:27.601562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.601500 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:27.872719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:27.872689 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm"] Sep 15 21:30:27.874247 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:27.874218 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507a859a_3273_48d9_9f34_458c24d8e34d.slice/crio-e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1 WatchSource:0}: Error finding container e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1: Status 404 returned error can't find the container with id e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1 Sep 15 21:30:28.018268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:28.018237 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" event={"ID":"507a859a-3273-48d9-9f34-458c24d8e34d","Type":"ContainerStarted","Data":"e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1"} Sep 15 21:30:29.025913 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:29.025878 2552 generic.go:356] "Generic (PLEG): container finished" podID="507a859a-3273-48d9-9f34-458c24d8e34d" containerID="06e9f266093067c7202a2b6a7eac8ffc570b829bb35b698930e1bd04b980b75d" exitCode=0 Sep 15 21:30:29.026309 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:29.025967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" event={"ID":"507a859a-3273-48d9-9f34-458c24d8e34d","Type":"ContainerDied","Data":"06e9f266093067c7202a2b6a7eac8ffc570b829bb35b698930e1bd04b980b75d"} Sep 15 21:30:30.166509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:30.166487 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:30.303655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:30.303577 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca\") pod \"507a859a-3273-48d9-9f34-458c24d8e34d\" (UID: \"507a859a-3273-48d9-9f34-458c24d8e34d\") " Sep 15 21:30:30.303937 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:30.303916 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "507a859a-3273-48d9-9f34-458c24d8e34d" (UID: "507a859a-3273-48d9-9f34-458c24d8e34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:30:30.404578 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:30.404543 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507a859a-3273-48d9-9f34-458c24d8e34d-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:30:31.038108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:31.038083 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" Sep 15 21:30:31.038264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:31.038124 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm" event={"ID":"507a859a-3273-48d9-9f34-458c24d8e34d","Type":"ContainerDied","Data":"e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1"} Sep 15 21:30:31.038264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:31.038169 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6e208265850f45e9995b1deb32298df54089e3b200ee54db3477417ef9888e1" Sep 15 21:30:34.262684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.262649 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:34.263164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.262913 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" podUID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" containerName="payload-processing" containerID="cri-o://416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f" gracePeriod=30 Sep 15 21:30:34.271538 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.271506 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:34.271740 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.271717 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" podUID="bb2d99af-ff58-4a94-8247-76675b37eeac" containerName="payload-pre-processing" containerID="cri-o://e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a" gracePeriod=30 Sep 15 21:30:34.444405 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.444368 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm"] Sep 15 21:30:34.446859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.446812 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a03be06-qggbm"] Sep 15 21:30:34.563804 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.563771 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:34.567487 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.567467 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:34.742772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.742742 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5rxc\" (UniqueName: \"kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc\") pod \"bb2d99af-ff58-4a94-8247-76675b37eeac\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " Sep 15 21:30:34.742772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.742781 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlwfs\" (UniqueName: \"kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs\") pod \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " Sep 15 21:30:34.743057 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.742983 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume\") pod \"bb2d99af-ff58-4a94-8247-76675b37eeac\" (UID: \"bb2d99af-ff58-4a94-8247-76675b37eeac\") " Sep 15 21:30:34.743117 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.743088 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume\") pod \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\" (UID: \"bc686c71-1158-4d62-a0c7-9cf501aeca7d\") " Sep 15 21:30:34.743403 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.743378 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb2d99af-ff58-4a94-8247-76675b37eeac" (UID: "bb2d99af-ff58-4a94-8247-76675b37eeac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:30:34.743495 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.743377 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc686c71-1158-4d62-a0c7-9cf501aeca7d" (UID: "bc686c71-1158-4d62-a0c7-9cf501aeca7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:30:34.744970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.744935 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc" (OuterVolumeSpecName: "kube-api-access-r5rxc") pod "bb2d99af-ff58-4a94-8247-76675b37eeac" (UID: "bb2d99af-ff58-4a94-8247-76675b37eeac"). InnerVolumeSpecName "kube-api-access-r5rxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:34.745081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.744971 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs" (OuterVolumeSpecName: "kube-api-access-jlwfs") pod "bc686c71-1158-4d62-a0c7-9cf501aeca7d" (UID: "bc686c71-1158-4d62-a0c7-9cf501aeca7d"). InnerVolumeSpecName "kube-api-access-jlwfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:34.844169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.844112 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rxc\" (UniqueName: \"kubernetes.io/projected/bb2d99af-ff58-4a94-8247-76675b37eeac-kube-api-access-r5rxc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:30:34.844169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.844134 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwfs\" (UniqueName: \"kubernetes.io/projected/bc686c71-1158-4d62-a0c7-9cf501aeca7d-kube-api-access-jlwfs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:30:34.844169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.844145 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2d99af-ff58-4a94-8247-76675b37eeac-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:30:34.844169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:34.844154 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc686c71-1158-4d62-a0c7-9cf501aeca7d-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:30:35.065952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.065920 2552 generic.go:356] "Generic (PLEG): container finished" podID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" containerID="416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f" exitCode=0 Sep 15 21:30:35.066148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.065971 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" Sep 15 21:30:35.066148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.066011 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" event={"ID":"bc686c71-1158-4d62-a0c7-9cf501aeca7d","Type":"ContainerDied","Data":"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f"} Sep 15 21:30:35.066148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.066073 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq" event={"ID":"bc686c71-1158-4d62-a0c7-9cf501aeca7d","Type":"ContainerDied","Data":"9f5ee3aa16433df03ca61a32f7235f97018835954c65c8b9cdf23588e72faf98"} Sep 15 21:30:35.066148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.066106 2552 scope.go:117] "RemoveContainer" containerID="416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f" Sep 15 21:30:35.067492 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.067474 2552 generic.go:356] "Generic (PLEG): container finished" podID="bb2d99af-ff58-4a94-8247-76675b37eeac" containerID="e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a" exitCode=0 Sep 15 21:30:35.067605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.067517 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" Sep 15 21:30:35.067605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.067532 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" event={"ID":"bb2d99af-ff58-4a94-8247-76675b37eeac","Type":"ContainerDied","Data":"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a"} Sep 15 21:30:35.067605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.067562 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp" event={"ID":"bb2d99af-ff58-4a94-8247-76675b37eeac","Type":"ContainerDied","Data":"c98848c0824fe83c529480707fb6717a4a9658876dc9a6b8c8503f92a4c72b7d"} Sep 15 21:30:35.083486 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.083470 2552 scope.go:117] "RemoveContainer" containerID="416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f" Sep 15 21:30:35.083763 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:30:35.083745 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f\": container with ID starting with 416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f not found: ID does not exist" containerID="416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f" Sep 15 21:30:35.083861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.083773 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f"} err="failed to get container status \"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f\": rpc error: code = NotFound desc = could not find container \"416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f\": container with ID starting with 416bccdf43e2a2124bfe37bd47cf5d8777892658481fa91732a04467271f7c5f not found: ID does not exist" Sep 15 21:30:35.083861 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.083795 2552 scope.go:117] "RemoveContainer" containerID="e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a" Sep 15 21:30:35.092272 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.092257 2552 scope.go:117] "RemoveContainer" containerID="e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a" Sep 15 21:30:35.092502 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:30:35.092485 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a\": container with ID starting with e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a not found: ID does not exist" containerID="e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a" Sep 15 21:30:35.092559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.092507 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a"} err="failed to get container status \"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a\": rpc error: code = NotFound desc = could not find container \"e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a\": container with ID starting with e61008c7e5814e209013ff2f3c6964f345ee1a8c359b380cea5ae16993f0933a not found: ID does not exist" Sep 15 21:30:35.097302 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.097254 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:35.104678 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.104656 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a03be06-559d8f5594-252hp"] Sep 15 21:30:35.124556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.124536 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:35.130561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.130540 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5a03be06-7777fd54d6-2qrkq"] Sep 15 21:30:35.346862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.346818 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:30:35.347449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347396 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507a859a-3273-48d9-9f34-458c24d8e34d" containerName="revoke-keys" Sep 15 21:30:35.347449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347419 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="507a859a-3273-48d9-9f34-458c24d8e34d" containerName="revoke-keys" Sep 15 21:30:35.347449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347431 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d99af-ff58-4a94-8247-76675b37eeac" containerName="payload-pre-processing" Sep 15 21:30:35.347449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347439 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d99af-ff58-4a94-8247-76675b37eeac" containerName="payload-pre-processing" Sep 15 21:30:35.347684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347452 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" containerName="payload-processing" Sep 15 21:30:35.347684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347461 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" containerName="payload-processing" Sep 15 21:30:35.347684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347579 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" containerName="payload-processing" Sep 15 21:30:35.347684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347594 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2d99af-ff58-4a94-8247-76675b37eeac" containerName="payload-pre-processing" Sep 15 21:30:35.347684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.347603 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="507a859a-3273-48d9-9f34-458c24d8e34d" containerName="revoke-keys" Sep 15 21:30:35.352126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.352106 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.356681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.356633 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-df1389fe-openshift-default-dockercfg-vth9g\"" Sep 15 21:30:35.366176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.366152 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:30:35.451952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.451921 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452006 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452114 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452161 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452293 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452216 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452293 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452253 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5kd\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452293 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452285 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452443 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452323 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.452443 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.452374 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553244 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553215 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553265 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553383 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553420 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553450 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5kd\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553482 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553519 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553748 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553560 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553748 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553617 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553748 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553636 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553748 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553676 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.553996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.553819 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.554054 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.554004 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.554325 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.554307 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.555459 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.555444 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.555907 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.555887 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.563446 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.563424 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.564404 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.564383 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5kd\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd\") pod \"e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.662361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.662339 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:35.790801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.790764 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="507a859a-3273-48d9-9f34-458c24d8e34d" path="/var/lib/kubelet/pods/507a859a-3273-48d9-9f34-458c24d8e34d/volumes" Sep 15 21:30:35.791222 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.791203 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2d99af-ff58-4a94-8247-76675b37eeac" path="/var/lib/kubelet/pods/bb2d99af-ff58-4a94-8247-76675b37eeac/volumes" Sep 15 21:30:35.791587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.791570 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc686c71-1158-4d62-a0c7-9cf501aeca7d" path="/var/lib/kubelet/pods/bc686c71-1158-4d62-a0c7-9cf501aeca7d/volumes" Sep 15 21:30:35.943128 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.943103 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:30:35.944892 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:35.944864 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc529afc3_4dbf_48e7_a6e3_fe45c1298919.slice/crio-fc011ce0891f2787581761284d773bf5d70bdcc5f66565cfa184a3dda6e5cf92 WatchSource:0}: Error finding container fc011ce0891f2787581761284d773bf5d70bdcc5f66565cfa184a3dda6e5cf92: Status 404 returned error can't find the container with id fc011ce0891f2787581761284d773bf5d70bdcc5f66565cfa184a3dda6e5cf92 Sep 15 21:30:35.947094 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.947063 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:35.947182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.947124 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:35.947182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:35.947153 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:36.076963 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:36.076858 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerStarted","Data":"2c99ed673e73ca4a784471b800c62dc0dda17b73c2c131b46ee25120188b820c"} Sep 15 21:30:36.076963 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:36.076928 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerStarted","Data":"fc011ce0891f2787581761284d773bf5d70bdcc5f66565cfa184a3dda6e5cf92"} Sep 15 21:30:36.099328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:36.099278 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podStartSLOduration=1.099262102 podStartE2EDuration="1.099262102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:36.096041161 +0000 UTC m=+876.731193271" watchObservedRunningTime="2026-09-15 21:30:36.099262102 +0000 UTC m=+876.734414213" Sep 15 21:30:36.662818 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:36.662788 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:37.682413 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:37.682332 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:38.091925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:38.091774 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:38.092681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:38.092663 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:41.003009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.002976 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:30:41.006697 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.006680 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.009496 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.009449 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df1389fe\"" Sep 15 21:30:41.009632 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.009539 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df1389fe-dockercfg-4qtvl\"" Sep 15 21:30:41.018249 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.018227 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:30:41.081268 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.081236 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:30:41.084796 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.084779 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.097590 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.097566 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:30:41.102430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.102409 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc69t\" (UniqueName: \"kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.102525 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.102437 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.102525 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.102458 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.102525 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.102511 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8cn\" (UniqueName: \"kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.203355 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.203322 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8cn\" (UniqueName: \"kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.203509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.203384 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc69t\" (UniqueName: \"kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.203509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.203402 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.203509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.203418 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.204078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.204061 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.204137 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.204125 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.215572 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.215550 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8cn\" (UniqueName: \"kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn\") pod \"payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.233433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.233411 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc69t\" (UniqueName: \"kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t\") pod \"payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.318334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.318264 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:41.396623 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.396320 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:41.595053 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.595023 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:30:41.596314 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:41.596290 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e36664_e5fe_4f9f_a958_6c10ac0efd2f.slice/crio-83074023d2d4ede4bdf2a47d665f8b1e0430fc833815678f57caa3edbee35d64 WatchSource:0}: Error finding container 83074023d2d4ede4bdf2a47d665f8b1e0430fc833815678f57caa3edbee35d64: Status 404 returned error can't find the container with id 83074023d2d4ede4bdf2a47d665f8b1e0430fc833815678f57caa3edbee35d64 Sep 15 21:30:41.676328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:41.676304 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:30:41.677075 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:30:41.677052 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2958b8d2_f234_45f0_9ba8_8e9388f4ff70.slice/crio-dd083f3ec1b369112d62c3a9c1fff4820939e5d52c8fdfe20f386993ba5aa806 WatchSource:0}: Error finding container dd083f3ec1b369112d62c3a9c1fff4820939e5d52c8fdfe20f386993ba5aa806: Status 404 returned error can't find the container with id dd083f3ec1b369112d62c3a9c1fff4820939e5d52c8fdfe20f386993ba5aa806 Sep 15 21:30:42.117155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.117117 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" event={"ID":"2958b8d2-f234-45f0-9ba8-8e9388f4ff70","Type":"ContainerStarted","Data":"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516"} Sep 15 21:30:42.123561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.117611 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:42.123561 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.117677 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" event={"ID":"2958b8d2-f234-45f0-9ba8-8e9388f4ff70","Type":"ContainerStarted","Data":"dd083f3ec1b369112d62c3a9c1fff4820939e5d52c8fdfe20f386993ba5aa806"} Sep 15 21:30:42.126037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.126007 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" event={"ID":"98e36664-e5fe-4f9f-a958-6c10ac0efd2f","Type":"ContainerStarted","Data":"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5"} Sep 15 21:30:42.126148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.126064 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" event={"ID":"98e36664-e5fe-4f9f-a958-6c10ac0efd2f","Type":"ContainerStarted","Data":"83074023d2d4ede4bdf2a47d665f8b1e0430fc833815678f57caa3edbee35d64"} Sep 15 21:30:42.126148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.126097 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:42.143243 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.143201 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" podStartSLOduration=1.143187346 podStartE2EDuration="1.143187346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:42.140069398 +0000 UTC m=+882.775221510" watchObservedRunningTime="2026-09-15 21:30:42.143187346 +0000 UTC m=+882.778339487" Sep 15 21:30:42.158432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:42.158387 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" podStartSLOduration=2.15837442 podStartE2EDuration="2.15837442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:42.157685845 +0000 UTC m=+882.792837956" watchObservedRunningTime="2026-09-15 21:30:42.15837442 +0000 UTC m=+882.793526556" Sep 15 21:30:53.135036 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:53.134961 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:30:53.135410 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:53.135039 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:30:57.225743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.225709 2552 generic.go:356] "Generic (PLEG): container finished" podID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerID="2c99ed673e73ca4a784471b800c62dc0dda17b73c2c131b46ee25120188b820c" exitCode=0 Sep 15 21:30:57.226178 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.225786 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerDied","Data":"2c99ed673e73ca4a784471b800c62dc0dda17b73c2c131b46ee25120188b820c"} Sep 15 21:30:57.226228 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.226170 2552 scope.go:117] "RemoveContainer" containerID="2c99ed673e73ca4a784471b800c62dc0dda17b73c2c131b46ee25120188b820c" Sep 15 21:30:57.227290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.227266 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:57.227361 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.227328 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:57.227401 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:57.227358 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:30:58.250243 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:58.249476 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerStarted","Data":"71c8590cedd1ecd81d50bdb1f685a876e779eabf3f5a08a5f3486ecbc1d97c61"} Sep 15 21:30:58.663313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:58.663277 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:30:58.663652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:58.663630 2552 patch_prober.go:28] interesting pod/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 15 21:30:58.663801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:58.663777 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 15 21:30:59.663131 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:59.663093 2552 patch_prober.go:28] interesting pod/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 15 21:30:59.663694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:59.663660 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 15 21:30:59.789046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:59.789008 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:30:59.791595 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:30:59.791572 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:31:00.663543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:00.663504 2552 patch_prober.go:28] interesting pod/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 15 21:31:00.664143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:00.664110 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 15 21:31:01.662886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:01.662789 2552 patch_prober.go:28] interesting pod/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 15 21:31:01.663196 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:01.663160 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 15 21:31:02.663351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:02.663314 2552 patch_prober.go:28] interesting pod/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 15 21:31:02.663986 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:02.663947 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 15 21:31:03.167476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.167435 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2"] Sep 15 21:31:03.171811 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.171787 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:03.174093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.174071 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:31:03.179145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.179122 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2"] Sep 15 21:31:03.303095 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.303064 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df1389fe-m85d2\" (UID: \"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:03.404403 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.404349 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df1389fe-m85d2\" (UID: \"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:03.405313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.405283 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df1389fe-m85d2\" (UID: \"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:03.485230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.485153 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:03.666894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.666869 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:31:03.667266 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.667248 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:31:03.667839 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.667804 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:31:03.766176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:03.766147 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2"] Sep 15 21:31:03.766341 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:31:03.766179 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97fdc3a_5cea_4dc2_8f70_8f1cea12678b.slice/crio-a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec WatchSource:0}: Error finding container a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec: Status 404 returned error can't find the container with id a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec Sep 15 21:31:04.295588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:04.295554 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" event={"ID":"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b","Type":"ContainerStarted","Data":"18d2e839eeb5d4b0ec93017e3c76ade276958b9309da45f79385c6e025efcf56"} Sep 15 21:31:04.295722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:04.295611 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" event={"ID":"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b","Type":"ContainerStarted","Data":"a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec"} Sep 15 21:31:04.310399 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:04.310319 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" podStartSLOduration=1.310304596 podStartE2EDuration="1.310304596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:04.308599387 +0000 UTC m=+904.943751501" watchObservedRunningTime="2026-09-15 21:31:04.310304596 +0000 UTC m=+904.945456706" Sep 15 21:31:05.309990 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:05.309955 2552 generic.go:356] "Generic (PLEG): container finished" podID="d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" containerID="18d2e839eeb5d4b0ec93017e3c76ade276958b9309da45f79385c6e025efcf56" exitCode=0 Sep 15 21:31:05.310402 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:05.310043 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" event={"ID":"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b","Type":"ContainerDied","Data":"18d2e839eeb5d4b0ec93017e3c76ade276958b9309da45f79385c6e025efcf56"} Sep 15 21:31:06.448224 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:06.448203 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:06.528673 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:06.528633 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca\") pod \"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b\" (UID: \"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b\") " Sep 15 21:31:06.528997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:06.528976 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" (UID: "d97fdc3a-5cea-4dc2-8f70-8f1cea12678b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:06.630148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:06.630114 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:07.322968 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:07.322936 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" Sep 15 21:31:07.323133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:07.322944 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2" event={"ID":"d97fdc3a-5cea-4dc2-8f70-8f1cea12678b","Type":"ContainerDied","Data":"a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec"} Sep 15 21:31:07.323133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:07.323055 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41b4464a4de207b043930446077b374b9c6bce9293e1e4e0cc11724126498ec" Sep 15 21:31:09.887894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:09.887851 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:31:09.888259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:09.888095 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" podUID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" containerName="payload-processing" containerID="cri-o://cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516" gracePeriod=30 Sep 15 21:31:09.896687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:09.896659 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:31:09.896904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:09.896874 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" podUID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" containerName="payload-pre-processing" containerID="cri-o://93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5" gracePeriod=30 Sep 15 21:31:10.071746 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.071711 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2"] Sep 15 21:31:10.082385 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.077291 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df1389fe-m85d2"] Sep 15 21:31:10.162735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.162711 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:31:10.179670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.179646 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:31:10.265305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265267 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc69t\" (UniqueName: \"kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t\") pod \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " Sep 15 21:31:10.265485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265319 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume\") pod \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\" (UID: \"2958b8d2-f234-45f0-9ba8-8e9388f4ff70\") " Sep 15 21:31:10.265485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265358 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume\") pod \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " Sep 15 21:31:10.265603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265529 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g8cn\" (UniqueName: \"kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn\") pod \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\" (UID: \"98e36664-e5fe-4f9f-a958-6c10ac0efd2f\") " Sep 15 21:31:10.265790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265764 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98e36664-e5fe-4f9f-a958-6c10ac0efd2f" (UID: "98e36664-e5fe-4f9f-a958-6c10ac0efd2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:10.265921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.265782 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2958b8d2-f234-45f0-9ba8-8e9388f4ff70" (UID: "2958b8d2-f234-45f0-9ba8-8e9388f4ff70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:10.266062 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.266037 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:10.266187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.266066 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:10.267499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.267472 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t" (OuterVolumeSpecName: "kube-api-access-bc69t") pod "2958b8d2-f234-45f0-9ba8-8e9388f4ff70" (UID: "2958b8d2-f234-45f0-9ba8-8e9388f4ff70"). InnerVolumeSpecName "kube-api-access-bc69t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:10.268140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.268067 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn" (OuterVolumeSpecName: "kube-api-access-7g8cn") pod "98e36664-e5fe-4f9f-a958-6c10ac0efd2f" (UID: "98e36664-e5fe-4f9f-a958-6c10ac0efd2f"). InnerVolumeSpecName "kube-api-access-7g8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:10.342164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.342126 2552 generic.go:356] "Generic (PLEG): container finished" podID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" containerID="93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5" exitCode=0 Sep 15 21:31:10.342298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.342170 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" Sep 15 21:31:10.342298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.342205 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" event={"ID":"98e36664-e5fe-4f9f-a958-6c10ac0efd2f","Type":"ContainerDied","Data":"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5"} Sep 15 21:31:10.342298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.342296 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg" event={"ID":"98e36664-e5fe-4f9f-a958-6c10ac0efd2f","Type":"ContainerDied","Data":"83074023d2d4ede4bdf2a47d665f8b1e0430fc833815678f57caa3edbee35d64"} Sep 15 21:31:10.342464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.342329 2552 scope.go:117] "RemoveContainer" containerID="93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5" Sep 15 21:31:10.343670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.343646 2552 generic.go:356] "Generic (PLEG): container finished" podID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" containerID="cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516" exitCode=0 Sep 15 21:31:10.343807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.343698 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" Sep 15 21:31:10.343807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.343712 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" event={"ID":"2958b8d2-f234-45f0-9ba8-8e9388f4ff70","Type":"ContainerDied","Data":"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516"} Sep 15 21:31:10.343807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.343743 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5" event={"ID":"2958b8d2-f234-45f0-9ba8-8e9388f4ff70","Type":"ContainerDied","Data":"dd083f3ec1b369112d62c3a9c1fff4820939e5d52c8fdfe20f386993ba5aa806"} Sep 15 21:31:10.352562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.352538 2552 scope.go:117] "RemoveContainer" containerID="93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5" Sep 15 21:31:10.352856 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:31:10.352811 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5\": container with ID starting with 93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5 not found: ID does not exist" containerID="93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5" Sep 15 21:31:10.352944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.352867 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5"} err="failed to get container status \"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5\": rpc error: code = NotFound desc = could not find container \"93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5\": container with ID starting with 93b18e3d1d4ea893f75cd330ca2f4ad3acf5ee2db32549fdc9a03869a96fa2c5 not found: ID does not exist" Sep 15 21:31:10.352944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.352890 2552 scope.go:117] "RemoveContainer" containerID="cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516" Sep 15 21:31:10.360893 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.360878 2552 scope.go:117] "RemoveContainer" containerID="cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516" Sep 15 21:31:10.361145 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:31:10.361106 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516\": container with ID starting with cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516 not found: ID does not exist" containerID="cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516" Sep 15 21:31:10.361145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.361126 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516"} err="failed to get container status \"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516\": rpc error: code = NotFound desc = could not find container \"cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516\": container with ID starting with cd0a407db9ae3d32ee23138035f523c5d4fd09d2205dee7765dc8db469c0e516 not found: ID does not exist" Sep 15 21:31:10.367145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.367118 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc69t\" (UniqueName: \"kubernetes.io/projected/2958b8d2-f234-45f0-9ba8-8e9388f4ff70-kube-api-access-bc69t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:10.367145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.367142 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8cn\" (UniqueName: \"kubernetes.io/projected/98e36664-e5fe-4f9f-a958-6c10ac0efd2f-kube-api-access-7g8cn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:10.368794 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.368774 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:31:10.373185 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.373167 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df1389fe-569ff5c6c-88lg5"] Sep 15 21:31:10.382257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.382240 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:31:10.386758 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:10.386738 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df1389fe-ff869c446-sxktg"] Sep 15 21:31:11.186077 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:11.185996 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:31:11.186512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:11.186198 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" containerID="cri-o://71c8590cedd1ecd81d50bdb1f685a876e779eabf3f5a08a5f3486ecbc1d97c61" gracePeriod=30 Sep 15 21:31:11.790478 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:11.790445 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" path="/var/lib/kubelet/pods/2958b8d2-f234-45f0-9ba8-8e9388f4ff70/volumes" Sep 15 21:31:11.790792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:11.790778 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" path="/var/lib/kubelet/pods/98e36664-e5fe-4f9f-a958-6c10ac0efd2f/volumes" Sep 15 21:31:11.791097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:11.791084 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" path="/var/lib/kubelet/pods/d97fdc3a-5cea-4dc2-8f70-8f1cea12678b/volumes" Sep 15 21:31:16.393664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.393636 2552 generic.go:356] "Generic (PLEG): container finished" podID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerID="71c8590cedd1ecd81d50bdb1f685a876e779eabf3f5a08a5f3486ecbc1d97c61" exitCode=0 Sep 15 21:31:16.394082 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.393710 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerDied","Data":"71c8590cedd1ecd81d50bdb1f685a876e779eabf3f5a08a5f3486ecbc1d97c61"} Sep 15 21:31:16.394082 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.393769 2552 scope.go:117] "RemoveContainer" containerID="2c99ed673e73ca4a784471b800c62dc0dda17b73c2c131b46ee25120188b820c" Sep 15 21:31:16.440102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.440081 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:31:16.521134 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521066 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521134 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521102 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521134 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521131 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521154 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521190 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521269 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521342 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521301 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521348 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521361 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:31:16.521542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521395 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb5kd\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd\") pod \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\" (UID: \"c529afc3-4dbf-48e7-a6e3-fe45c1298919\") " Sep 15 21:31:16.521542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521415 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:31:16.521542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521523 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:31:16.521845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521625 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:16.521845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521761 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data" (OuterVolumeSpecName: "istio-data") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:31:16.521970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521892 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.521970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521915 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.521970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521929 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.521970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521943 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.521970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.521956 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.523287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.523263 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:31:16.523445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.523410 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd" (OuterVolumeSpecName: "kube-api-access-sb5kd") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "kube-api-access-sb5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:16.523539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.523453 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:31:16.523539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.523519 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token" (OuterVolumeSpecName: "istio-token") pod "c529afc3-4dbf-48e7-a6e3-fe45c1298919" (UID: "c529afc3-4dbf-48e7-a6e3-fe45c1298919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:16.623066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.623041 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.623066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.623063 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.623213 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.623072 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5kd\" (UniqueName: \"kubernetes.io/projected/c529afc3-4dbf-48e7-a6e3-fe45c1298919-kube-api-access-sb5kd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:16.623213 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:16.623083 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c529afc3-4dbf-48e7-a6e3-fe45c1298919-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:17.401773 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.401745 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" Sep 15 21:31:17.402178 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.401782 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj" event={"ID":"c529afc3-4dbf-48e7-a6e3-fe45c1298919","Type":"ContainerDied","Data":"fc011ce0891f2787581761284d773bf5d70bdcc5f66565cfa184a3dda6e5cf92"} Sep 15 21:31:17.402178 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.401849 2552 scope.go:117] "RemoveContainer" containerID="71c8590cedd1ecd81d50bdb1f685a876e779eabf3f5a08a5f3486ecbc1d97c61" Sep 15 21:31:17.427979 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.427950 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:31:17.431764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.431743 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-df1389fe-openshift-default-5f4b54f9b8-wmqdj"] Sep 15 21:31:17.790807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:17.790722 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" path="/var/lib/kubelet/pods/c529afc3-4dbf-48e7-a6e3-fe45c1298919/volumes" Sep 15 21:31:21.112003 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.111967 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112339 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112351 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112359 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" containerName="payload-pre-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112365 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" containerName="payload-pre-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112379 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" containerName="payload-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112385 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" containerName="payload-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112395 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" containerName="revoke-keys" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112401 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" containerName="revoke-keys" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112468 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958b8d2-f234-45f0-9ba8-8e9388f4ff70" containerName="payload-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112479 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97fdc3a-5cea-4dc2-8f70-8f1cea12678b" containerName="revoke-keys" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112487 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112493 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e36664-e5fe-4f9f-a958-6c10ac0efd2f" containerName="payload-pre-processing" Sep 15 21:31:21.112497 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.112498 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.117004 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.116989 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.119110 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.119086 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d578a96c\"" Sep 15 21:31:21.119230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.119141 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d578a96c-dockercfg-k4vtx\"" Sep 15 21:31:21.127672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.127647 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:31:21.151840 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.151794 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:31:21.152225 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.152208 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.152277 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.152227 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c529afc3-4dbf-48e7-a6e3-fe45c1298919" containerName="istio-proxy" Sep 15 21:31:21.155064 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.155049 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.164909 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.164870 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:31:21.263306 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.263274 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.263473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.263316 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.263473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.263337 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nc7\" (UniqueName: \"kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.263473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.263414 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbn4\" (UniqueName: \"kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.364802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.364719 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbn4\" (UniqueName: \"kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.364802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.364796 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.365050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.364871 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.365050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.364900 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nc7\" (UniqueName: \"kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.365720 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.365685 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.365720 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.365715 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.372733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.372705 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbn4\" (UniqueName: \"kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4\") pod \"payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.372821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.372717 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nc7\" (UniqueName: \"kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7\") pod \"payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.427869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.427819 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:21.466351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.466312 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:21.700714 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.700692 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:31:21.702896 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:31:21.702864 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359c5bf1_175d_48c6_8b47_3f16cc6590f8.slice/crio-1c59994222bd53ca3ed8dde0e09b9e07a052e92e88fb37b5ad8358b9d2582eac WatchSource:0}: Error finding container 1c59994222bd53ca3ed8dde0e09b9e07a052e92e88fb37b5ad8358b9d2582eac: Status 404 returned error can't find the container with id 1c59994222bd53ca3ed8dde0e09b9e07a052e92e88fb37b5ad8358b9d2582eac Sep 15 21:31:21.755644 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:21.755619 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:31:21.757591 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:31:21.757570 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4530419b_6d30_422c_96b7_5d83bdd6250d.slice/crio-e9a7d57ec4c1b6a43d00a657cec56b6b03f1738bb533a3944e3b8dcd3f78dfd7 WatchSource:0}: Error finding container e9a7d57ec4c1b6a43d00a657cec56b6b03f1738bb533a3944e3b8dcd3f78dfd7: Status 404 returned error can't find the container with id e9a7d57ec4c1b6a43d00a657cec56b6b03f1738bb533a3944e3b8dcd3f78dfd7 Sep 15 21:31:22.434028 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.433990 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" event={"ID":"359c5bf1-175d-48c6-8b47-3f16cc6590f8","Type":"ContainerStarted","Data":"f18ba387a6805dd7349cfee1a4fb1c44e3447d90421527d91b4cd7ebbfc0e226"} Sep 15 21:31:22.434453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.434043 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" event={"ID":"359c5bf1-175d-48c6-8b47-3f16cc6590f8","Type":"ContainerStarted","Data":"1c59994222bd53ca3ed8dde0e09b9e07a052e92e88fb37b5ad8358b9d2582eac"} Sep 15 21:31:22.434453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.434204 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:22.435596 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.435569 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" event={"ID":"4530419b-6d30-422c-96b7-5d83bdd6250d","Type":"ContainerStarted","Data":"fa01b9af1633d6937b656c776b01afc9edda6a30e1c5b5455444caeaa550d864"} Sep 15 21:31:22.435725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.435608 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" event={"ID":"4530419b-6d30-422c-96b7-5d83bdd6250d","Type":"ContainerStarted","Data":"e9a7d57ec4c1b6a43d00a657cec56b6b03f1738bb533a3944e3b8dcd3f78dfd7"} Sep 15 21:31:22.435725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.435634 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:22.453194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.453143 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" podStartSLOduration=1.453130586 podStartE2EDuration="1.453130586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:22.449911054 +0000 UTC m=+923.085063165" watchObservedRunningTime="2026-09-15 21:31:22.453130586 +0000 UTC m=+923.088282696" Sep 15 21:31:22.470137 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:22.470098 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" podStartSLOduration=1.4700860869999999 podStartE2EDuration="1.470086087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:22.467383295 +0000 UTC m=+923.102535407" watchObservedRunningTime="2026-09-15 21:31:22.470086087 +0000 UTC m=+923.105238197" Sep 15 21:31:33.450689 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:33.450657 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:33.451066 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:33.450843 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:44.892454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:44.892417 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt"] Sep 15 21:31:44.896063 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:44.896045 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:44.898040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:44.898019 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:31:44.903097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:44.903075 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt"] Sep 15 21:31:44.951585 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:44.951557 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d578a96c-t64rt\" (UID: \"eb909e4e-5849-4575-b476-23975404546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:45.052202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:45.052173 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d578a96c-t64rt\" (UID: \"eb909e4e-5849-4575-b476-23975404546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:45.052745 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:45.052727 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d578a96c-t64rt\" (UID: \"eb909e4e-5849-4575-b476-23975404546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:45.208881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:45.208812 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:45.486353 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:45.486325 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt"] Sep 15 21:31:45.487568 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:31:45.487540 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb909e4e_5849_4575_b476_23975404546a.slice/crio-b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c WatchSource:0}: Error finding container b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c: Status 404 returned error can't find the container with id b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c Sep 15 21:31:45.578128 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:45.578094 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" event={"ID":"eb909e4e-5849-4575-b476-23975404546a","Type":"ContainerStarted","Data":"b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c"} Sep 15 21:31:46.586042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:46.586011 2552 generic.go:356] "Generic (PLEG): container finished" podID="eb909e4e-5849-4575-b476-23975404546a" containerID="658589c6d68e8a052f11083dd7153c7d666f94fdf0aa76ca66ec9a854f17702c" exitCode=0 Sep 15 21:31:46.586468 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:46.586065 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" event={"ID":"eb909e4e-5849-4575-b476-23975404546a","Type":"ContainerDied","Data":"658589c6d68e8a052f11083dd7153c7d666f94fdf0aa76ca66ec9a854f17702c"} Sep 15 21:31:47.725801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:47.725780 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:47.775469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:47.775444 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca\") pod \"eb909e4e-5849-4575-b476-23975404546a\" (UID: \"eb909e4e-5849-4575-b476-23975404546a\") " Sep 15 21:31:47.775765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:47.775745 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb909e4e-5849-4575-b476-23975404546a" (UID: "eb909e4e-5849-4575-b476-23975404546a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:47.876166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:47.876139 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb909e4e-5849-4575-b476-23975404546a-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:48.599877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:48.599839 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" event={"ID":"eb909e4e-5849-4575-b476-23975404546a","Type":"ContainerDied","Data":"b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c"} Sep 15 21:31:48.599877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:48.599864 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt" Sep 15 21:31:48.600123 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:48.599926 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b857c24b26ef01d84ea84835ccbc9ca87aedc0d38c4c63aef3003b1a7153182c" Sep 15 21:31:54.786909 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.786874 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4"] Sep 15 21:31:54.787411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.787393 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb909e4e-5849-4575-b476-23975404546a" containerName="revoke-keys" Sep 15 21:31:54.787458 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.787415 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb909e4e-5849-4575-b476-23975404546a" containerName="revoke-keys" Sep 15 21:31:54.787518 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.787507 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb909e4e-5849-4575-b476-23975404546a" containerName="revoke-keys" Sep 15 21:31:54.790735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.790718 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:54.792886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.792871 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:31:54.798015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.797991 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4"] Sep 15 21:31:54.934744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:54.934712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-10c104-48ln4\" (UID: \"2ea32c57-b46c-416f-ad3c-d1ec47ccd790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:55.036149 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:55.036123 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-10c104-48ln4\" (UID: \"2ea32c57-b46c-416f-ad3c-d1ec47ccd790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:55.036763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:55.036743 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-10c104-48ln4\" (UID: \"2ea32c57-b46c-416f-ad3c-d1ec47ccd790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:55.103170 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:55.103098 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:55.380349 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:55.380321 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4"] Sep 15 21:31:55.382029 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:31:55.382001 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea32c57_b46c_416f_ad3c_d1ec47ccd790.slice/crio-47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b WatchSource:0}: Error finding container 47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b: Status 404 returned error can't find the container with id 47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b Sep 15 21:31:55.643560 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:55.643528 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" event={"ID":"2ea32c57-b46c-416f-ad3c-d1ec47ccd790","Type":"ContainerStarted","Data":"47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b"} Sep 15 21:31:56.651277 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:56.651245 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ea32c57-b46c-416f-ad3c-d1ec47ccd790" containerID="4e2c94b1bec640ef2aff431d886574ca86e2ec7ff77973d61439ab28f9a67b92" exitCode=0 Sep 15 21:31:56.651729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:56.651330 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" event={"ID":"2ea32c57-b46c-416f-ad3c-d1ec47ccd790","Type":"ContainerDied","Data":"4e2c94b1bec640ef2aff431d886574ca86e2ec7ff77973d61439ab28f9a67b92"} Sep 15 21:31:57.797135 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:57.797113 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:57.862950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:57.862915 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca\") pod \"2ea32c57-b46c-416f-ad3c-d1ec47ccd790\" (UID: \"2ea32c57-b46c-416f-ad3c-d1ec47ccd790\") " Sep 15 21:31:57.863230 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:57.863210 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ea32c57-b46c-416f-ad3c-d1ec47ccd790" (UID: "2ea32c57-b46c-416f-ad3c-d1ec47ccd790"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:57.964145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:57.964075 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea32c57-b46c-416f-ad3c-d1ec47ccd790-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:58.665232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:58.665203 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" Sep 15 21:31:58.665391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:58.665198 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4" event={"ID":"2ea32c57-b46c-416f-ad3c-d1ec47ccd790","Type":"ContainerDied","Data":"47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b"} Sep 15 21:31:58.665391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:58.665328 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a0947eee55f40c79e482b7cd60859c921bb3c1a5c3aaccdc477fda2922b54b" Sep 15 21:31:59.438515 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.438485 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:31:59.439002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.438695 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" podUID="4530419b-6d30-422c-96b7-5d83bdd6250d" containerName="payload-processing" containerID="cri-o://fa01b9af1633d6937b656c776b01afc9edda6a30e1c5b5455444caeaa550d864" gracePeriod=30 Sep 15 21:31:59.448784 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.448742 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:31:59.449014 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.448978 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" podUID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" containerName="payload-pre-processing" containerID="cri-o://f18ba387a6805dd7349cfee1a4fb1c44e3447d90421527d91b4cd7ebbfc0e226" gracePeriod=30 Sep 15 21:31:59.624114 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.624074 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt"] Sep 15 21:31:59.626625 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.626597 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d578a96c-t64rt"] Sep 15 21:31:59.676304 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.676284 2552 generic.go:356] "Generic (PLEG): container finished" podID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" containerID="f18ba387a6805dd7349cfee1a4fb1c44e3447d90421527d91b4cd7ebbfc0e226" exitCode=0 Sep 15 21:31:59.676395 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.676366 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" event={"ID":"359c5bf1-175d-48c6-8b47-3f16cc6590f8","Type":"ContainerDied","Data":"f18ba387a6805dd7349cfee1a4fb1c44e3447d90421527d91b4cd7ebbfc0e226"} Sep 15 21:31:59.678629 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.678601 2552 generic.go:356] "Generic (PLEG): container finished" podID="4530419b-6d30-422c-96b7-5d83bdd6250d" containerID="fa01b9af1633d6937b656c776b01afc9edda6a30e1c5b5455444caeaa550d864" exitCode=0 Sep 15 21:31:59.678758 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.678651 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" event={"ID":"4530419b-6d30-422c-96b7-5d83bdd6250d","Type":"ContainerDied","Data":"fa01b9af1633d6937b656c776b01afc9edda6a30e1c5b5455444caeaa550d864"} Sep 15 21:31:59.698608 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.698550 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:31:59.727350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.727326 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:31:59.779287 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.779261 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume\") pod \"4530419b-6d30-422c-96b7-5d83bdd6250d\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " Sep 15 21:31:59.779459 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.779314 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7nc7\" (UniqueName: \"kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7\") pod \"4530419b-6d30-422c-96b7-5d83bdd6250d\" (UID: \"4530419b-6d30-422c-96b7-5d83bdd6250d\") " Sep 15 21:31:59.779664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.779642 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4530419b-6d30-422c-96b7-5d83bdd6250d" (UID: "4530419b-6d30-422c-96b7-5d83bdd6250d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:59.782515 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.782490 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7" (OuterVolumeSpecName: "kube-api-access-m7nc7") pod "4530419b-6d30-422c-96b7-5d83bdd6250d" (UID: "4530419b-6d30-422c-96b7-5d83bdd6250d"). InnerVolumeSpecName "kube-api-access-m7nc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:59.792352 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.792327 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb909e4e-5849-4575-b476-23975404546a" path="/var/lib/kubelet/pods/eb909e4e-5849-4575-b476-23975404546a/volumes" Sep 15 21:31:59.880532 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.880489 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume\") pod \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " Sep 15 21:31:59.880716 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.880540 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgbn4\" (UniqueName: \"kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4\") pod \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\" (UID: \"359c5bf1-175d-48c6-8b47-3f16cc6590f8\") " Sep 15 21:31:59.880950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.880900 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7nc7\" (UniqueName: \"kubernetes.io/projected/4530419b-6d30-422c-96b7-5d83bdd6250d-kube-api-access-m7nc7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.880950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.880924 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4530419b-6d30-422c-96b7-5d83bdd6250d-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.881112 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.880978 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "359c5bf1-175d-48c6-8b47-3f16cc6590f8" (UID: "359c5bf1-175d-48c6-8b47-3f16cc6590f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:59.883234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.883211 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4" (OuterVolumeSpecName: "kube-api-access-lgbn4") pod "359c5bf1-175d-48c6-8b47-3f16cc6590f8" (UID: "359c5bf1-175d-48c6-8b47-3f16cc6590f8"). InnerVolumeSpecName "kube-api-access-lgbn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:59.981967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.981892 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359c5bf1-175d-48c6-8b47-3f16cc6590f8-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.981967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:31:59.981928 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgbn4\" (UniqueName: \"kubernetes.io/projected/359c5bf1-175d-48c6-8b47-3f16cc6590f8-kube-api-access-lgbn4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:32:00.689896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.689861 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" event={"ID":"359c5bf1-175d-48c6-8b47-3f16cc6590f8","Type":"ContainerDied","Data":"1c59994222bd53ca3ed8dde0e09b9e07a052e92e88fb37b5ad8358b9d2582eac"} Sep 15 21:32:00.689896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.689891 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l" Sep 15 21:32:00.690397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.689929 2552 scope.go:117] "RemoveContainer" containerID="f18ba387a6805dd7349cfee1a4fb1c44e3447d90421527d91b4cd7ebbfc0e226" Sep 15 21:32:00.691355 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.691337 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" Sep 15 21:32:00.691479 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.691349 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd" event={"ID":"4530419b-6d30-422c-96b7-5d83bdd6250d","Type":"ContainerDied","Data":"e9a7d57ec4c1b6a43d00a657cec56b6b03f1738bb533a3944e3b8dcd3f78dfd7"} Sep 15 21:32:00.701050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.701034 2552 scope.go:117] "RemoveContainer" containerID="fa01b9af1633d6937b656c776b01afc9edda6a30e1c5b5455444caeaa550d864" Sep 15 21:32:00.713894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.713872 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:32:00.721179 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.721158 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d578a96c-58886f55b9-k59cd"] Sep 15 21:32:00.738526 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.738505 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:32:00.742772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:00.742752 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d578a96c-7776659bd6-jhg7l"] Sep 15 21:32:01.790097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:01.790065 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" path="/var/lib/kubelet/pods/359c5bf1-175d-48c6-8b47-3f16cc6590f8/volumes" Sep 15 21:32:01.790449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:01.790386 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4530419b-6d30-422c-96b7-5d83bdd6250d" path="/var/lib/kubelet/pods/4530419b-6d30-422c-96b7-5d83bdd6250d/volumes" Sep 15 21:32:06.425996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.425964 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln"] Sep 15 21:32:06.435572 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.435543 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-10c104-29825130-lqpln"] Sep 15 21:32:06.498867 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.498813 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:32:06.499103 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.499071 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" podUID="f4f55973-3f5a-4d37-933d-cb509168fbf8" containerName="payload-pre-processing" containerID="cri-o://2a3839a0c37da318c7ebc6c68bca9222637e69148850c10de3c0684ecd18114a" gracePeriod=30 Sep 15 21:32:06.673247 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.673215 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4"] Sep 15 21:32:06.675729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.675708 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-10c104-48ln4"] Sep 15 21:32:06.734012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.733929 2552 generic.go:356] "Generic (PLEG): container finished" podID="f4f55973-3f5a-4d37-933d-cb509168fbf8" containerID="2a3839a0c37da318c7ebc6c68bca9222637e69148850c10de3c0684ecd18114a" exitCode=0 Sep 15 21:32:06.734143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.734000 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" event={"ID":"f4f55973-3f5a-4d37-933d-cb509168fbf8","Type":"ContainerDied","Data":"2a3839a0c37da318c7ebc6c68bca9222637e69148850c10de3c0684ecd18114a"} Sep 15 21:32:06.761204 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.761180 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:32:06.837545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.837516 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume\") pod \"f4f55973-3f5a-4d37-933d-cb509168fbf8\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " Sep 15 21:32:06.837545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.837548 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz5cd\" (UniqueName: \"kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd\") pod \"f4f55973-3f5a-4d37-933d-cb509168fbf8\" (UID: \"f4f55973-3f5a-4d37-933d-cb509168fbf8\") " Sep 15 21:32:06.837974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.837948 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4f55973-3f5a-4d37-933d-cb509168fbf8" (UID: "f4f55973-3f5a-4d37-933d-cb509168fbf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:32:06.839556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.839535 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd" (OuterVolumeSpecName: "kube-api-access-kz5cd") pod "f4f55973-3f5a-4d37-933d-cb509168fbf8" (UID: "f4f55973-3f5a-4d37-933d-cb509168fbf8"). InnerVolumeSpecName "kube-api-access-kz5cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:32:06.939033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.938957 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f55973-3f5a-4d37-933d-cb509168fbf8-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:32:06.939033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:06.938989 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz5cd\" (UniqueName: \"kubernetes.io/projected/f4f55973-3f5a-4d37-933d-cb509168fbf8-kube-api-access-kz5cd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:32:07.742366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.742306 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" event={"ID":"f4f55973-3f5a-4d37-933d-cb509168fbf8","Type":"ContainerDied","Data":"081a5b67575c7c7131de3be72e9bfb3a93d1e5e18f070eca6f470369fc8ca615"} Sep 15 21:32:07.743057 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.743032 2552 scope.go:117] "RemoveContainer" containerID="2a3839a0c37da318c7ebc6c68bca9222637e69148850c10de3c0684ecd18114a" Sep 15 21:32:07.743432 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.743413 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9" Sep 15 21:32:07.775809 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.775784 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:32:07.790789 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.790760 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea32c57-b46c-416f-ad3c-d1ec47ccd790" path="/var/lib/kubelet/pods/2ea32c57-b46c-416f-ad3c-d1ec47ccd790/volumes" Sep 15 21:32:07.791119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.791104 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf7cdca-e0e9-4803-826a-69f3e62f4e17" path="/var/lib/kubelet/pods/bdf7cdca-e0e9-4803-826a-69f3e62f4e17/volumes" Sep 15 21:32:07.791336 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:07.791323 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-10c104-888747cbb-258k9"] Sep 15 21:32:09.791662 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:09.791629 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f55973-3f5a-4d37-933d-cb509168fbf8" path="/var/lib/kubelet/pods/f4f55973-3f5a-4d37-933d-cb509168fbf8/volumes" Sep 15 21:32:41.716789 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.716747 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:32:41.717365 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717343 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" containerName="payload-pre-processing" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717368 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" containerName="payload-pre-processing" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717391 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea32c57-b46c-416f-ad3c-d1ec47ccd790" containerName="revoke-keys" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717400 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea32c57-b46c-416f-ad3c-d1ec47ccd790" containerName="revoke-keys" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717416 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f55973-3f5a-4d37-933d-cb509168fbf8" containerName="payload-pre-processing" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717425 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f55973-3f5a-4d37-933d-cb509168fbf8" containerName="payload-pre-processing" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717437 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4530419b-6d30-422c-96b7-5d83bdd6250d" containerName="payload-processing" Sep 15 21:32:41.717455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717446 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4530419b-6d30-422c-96b7-5d83bdd6250d" containerName="payload-processing" Sep 15 21:32:41.717801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717552 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea32c57-b46c-416f-ad3c-d1ec47ccd790" containerName="revoke-keys" Sep 15 21:32:41.717801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717567 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="359c5bf1-175d-48c6-8b47-3f16cc6590f8" containerName="payload-pre-processing" Sep 15 21:32:41.717801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717578 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4530419b-6d30-422c-96b7-5d83bdd6250d" containerName="payload-processing" Sep 15 21:32:41.717801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.717589 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f55973-3f5a-4d37-933d-cb509168fbf8" containerName="payload-pre-processing" Sep 15 21:32:41.720313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.720294 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.723668 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.723637 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"openshift-service-ca.crt\"" Sep 15 21:32:41.723786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.723637 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"default-dockercfg-g8wsl\"" Sep 15 21:32:41.723786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.723683 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"kube-root-ca.crt\"" Sep 15 21:32:41.723786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.723642 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"premium-w0-b99d7b-kserve-self-signed-certs\"" Sep 15 21:32:41.732529 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.732507 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.819767 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.819852 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.819885 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx67b\" (UniqueName: \"kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.819988 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.820050 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.820587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.820162 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.920680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.920650 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.920898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.920723 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.920898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.920764 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.920898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.920799 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.920898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.920840 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx67b\" (UniqueName: \"kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.921119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.921024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.921183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.921137 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.921183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.921172 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.921334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.921315 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.923046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.923023 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.923476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.923458 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:41.928775 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:41.928754 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx67b\" (UniqueName: \"kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b\") pod \"premium-w0-b99d7b-kserve-6b47885795-dlxvk\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:42.030101 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:42.030031 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:42.306332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:42.306304 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:32:42.307458 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:32:42.307429 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf51657fa_fdd7_4fdd_b5e2_beec15482a55.slice/crio-7e1c5343c07a3acd3159bf013d27b44f3d8408926dc3fe41816d8362a4cdf638 WatchSource:0}: Error finding container 7e1c5343c07a3acd3159bf013d27b44f3d8408926dc3fe41816d8362a4cdf638: Status 404 returned error can't find the container with id 7e1c5343c07a3acd3159bf013d27b44f3d8408926dc3fe41816d8362a4cdf638 Sep 15 21:32:42.950181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:42.950146 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" event={"ID":"f51657fa-fdd7-4fdd-b5e2-beec15482a55","Type":"ContainerStarted","Data":"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88"} Sep 15 21:32:42.950543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:42.950195 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" event={"ID":"f51657fa-fdd7-4fdd-b5e2-beec15482a55","Type":"ContainerStarted","Data":"7e1c5343c07a3acd3159bf013d27b44f3d8408926dc3fe41816d8362a4cdf638"} Sep 15 21:32:42.967700 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:42.967652 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" podStartSLOduration=1.9676356259999999 podStartE2EDuration="1.967635626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:42.966047869 +0000 UTC m=+1003.601199983" watchObservedRunningTime="2026-09-15 21:32:42.967635626 +0000 UTC m=+1003.602787737" Sep 15 21:32:50.096675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.096638 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:32:50.100503 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.100483 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.102754 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.102731 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-32270d-openshift-default-dockercfg-6md4r\"" Sep 15 21:32:50.114789 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.114766 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:32:50.202965 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.202932 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.202973 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203013 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203120 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203170 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203206 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203306 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203332 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vpw\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.203542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.203383 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304055 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304089 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vpw\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304114 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304232 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304288 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304358 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304384 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304409 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304425 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304485 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304761 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304546 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304823 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304792 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304931 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304853 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.304991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.304939 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.306690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.306669 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.306795 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.306722 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.312004 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.311980 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.312734 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.312718 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vpw\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw\") pod \"e2e-explicit-32270d-openshift-default-798c96bdcb-95m94\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.411266 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.411245 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:50.687930 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.687904 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:32:50.688970 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:32:50.688944 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57d90e9_9334_42eb_98b3_083150ed691c.slice/crio-9656f249db57a1b17aabfa103dd9b25f040b61db1d960c46f667f9db5e66a1e9 WatchSource:0}: Error finding container 9656f249db57a1b17aabfa103dd9b25f040b61db1d960c46f667f9db5e66a1e9: Status 404 returned error can't find the container with id 9656f249db57a1b17aabfa103dd9b25f040b61db1d960c46f667f9db5e66a1e9 Sep 15 21:32:50.691491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.691460 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:32:50.691568 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.691531 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:32:50.691568 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.691559 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:32:50.999734 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.999653 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerStarted","Data":"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928"} Sep 15 21:32:50.999734 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:50.999713 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerStarted","Data":"9656f249db57a1b17aabfa103dd9b25f040b61db1d960c46f667f9db5e66a1e9"} Sep 15 21:32:51.020321 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:51.020270 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podStartSLOduration=1.020249358 podStartE2EDuration="1.020249358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:51.017755498 +0000 UTC m=+1011.652907606" watchObservedRunningTime="2026-09-15 21:32:51.020249358 +0000 UTC m=+1011.655401467" Sep 15 21:32:51.411491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:51.411459 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:51.416398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:51.416376 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:52.005997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:52.005967 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:52.006932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:52.006908 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:32:52.030803 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:52.030776 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:52.030927 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:52.030814 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:52.044246 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:52.044224 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:32:53.023548 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:32:53.023515 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:33:21.210976 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.210943 2552 generic.go:356] "Generic (PLEG): container finished" podID="b57d90e9-9334-42eb-98b3-083150ed691c" containerID="aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928" exitCode=0 Sep 15 21:33:21.211382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.211011 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerDied","Data":"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928"} Sep 15 21:33:21.211449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.211430 2552 scope.go:117] "RemoveContainer" containerID="aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928" Sep 15 21:33:21.212648 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.212617 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:21.212767 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.212702 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:21.212767 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:21.212751 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:22.006869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:22.006819 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:33:22.218723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:22.218689 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerStarted","Data":"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2"} Sep 15 21:33:22.411418 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:22.411388 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:33:22.412822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:22.412790 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:22.412974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:22.412866 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:23.412623 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:23.412585 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:23.413150 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:23.412662 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:24.411957 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:24.411924 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:24.412149 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:24.411984 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:25.411989 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:25.411953 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:25.412372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:25.412006 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:26.411763 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:26.411729 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:26.411959 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:26.411782 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:27.411591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:27.411558 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 15 21:33:27.411981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:27.411609 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 15 21:33:28.416026 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:28.415994 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:33:28.416382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:28.416286 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:33:28.416898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:28.416878 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:33:35.959002 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.958967 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:33:35.962643 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.962619 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.964745 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.964724 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1664eb4-openshift-default-dockercfg-gl4rl\"" Sep 15 21:33:35.973094 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.973071 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:33:35.994285 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994258 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994306 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994375 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwsg\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994412 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994447 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994466 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994578 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994652 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:35.994894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:35.994706 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.095907 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.095876 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.095933 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.095961 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.095986 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096014 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096091 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwsg\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096145 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096195 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096221 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096288 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096276 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096538 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096309 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096538 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096343 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.096719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.096698 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.097031 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.097010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.098372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.098350 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.098468 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.098453 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.104001 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.103977 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwsg\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.104233 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.104217 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token\") pod \"e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.274372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.274288 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:36.641497 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:33:36.641462 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e274bbb_d170_45e3_bfdc_e6629139b75f.slice/crio-d2fc7f9ea25a1973183531ad6605b5d4b499f3b3eb0948c67e5d5f64c913e09c WatchSource:0}: Error finding container d2fc7f9ea25a1973183531ad6605b5d4b499f3b3eb0948c67e5d5f64c913e09c: Status 404 returned error can't find the container with id d2fc7f9ea25a1973183531ad6605b5d4b499f3b3eb0948c67e5d5f64c913e09c Sep 15 21:33:36.644211 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.644173 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:36.644292 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.644262 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:36.644330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.644302 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:33:36.644858 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:36.644823 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:33:37.321913 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:37.321869 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerStarted","Data":"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707"} Sep 15 21:33:37.322278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:37.321930 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerStarted","Data":"d2fc7f9ea25a1973183531ad6605b5d4b499f3b3eb0948c67e5d5f64c913e09c"} Sep 15 21:33:37.344791 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:37.344748 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podStartSLOduration=2.344735025 podStartE2EDuration="2.344735025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:37.343557544 +0000 UTC m=+1057.978709645" watchObservedRunningTime="2026-09-15 21:33:37.344735025 +0000 UTC m=+1057.979887135" Sep 15 21:33:38.274499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:38.274460 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:38.279444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:38.279418 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:38.328573 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:38.328543 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:38.329252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:38.329236 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:33:46.624551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.624517 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:33:46.627275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.627259 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.629522 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.629507 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1664eb4-dockercfg-682b5\"" Sep 15 21:33:46.629581 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.629542 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1664eb4\"" Sep 15 21:33:46.638305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.638282 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:33:46.685181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.685138 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.685375 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.685312 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwf8\" (UniqueName: \"kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.786207 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.786178 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwf8\" (UniqueName: \"kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.786371 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.786230 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.786960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.786935 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.799139 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.799114 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwf8\" (UniqueName: \"kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8\") pod \"payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:46.939505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:46.939449 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:47.215940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:47.215914 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:33:47.217048 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:33:47.217018 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01af4df7_ecdf_4086_af11_3fd6c08e1a37.slice/crio-5ac98f1562ba624c08a95c63a555237fb70afd5f5b3db5b327b4958eafa2368b WatchSource:0}: Error finding container 5ac98f1562ba624c08a95c63a555237fb70afd5f5b3db5b327b4958eafa2368b: Status 404 returned error can't find the container with id 5ac98f1562ba624c08a95c63a555237fb70afd5f5b3db5b327b4958eafa2368b Sep 15 21:33:47.387037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:47.387000 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" event={"ID":"01af4df7-ecdf-4086-af11-3fd6c08e1a37","Type":"ContainerStarted","Data":"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82"} Sep 15 21:33:47.387184 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:47.387057 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" event={"ID":"01af4df7-ecdf-4086-af11-3fd6c08e1a37","Type":"ContainerStarted","Data":"5ac98f1562ba624c08a95c63a555237fb70afd5f5b3db5b327b4958eafa2368b"} Sep 15 21:33:47.387184 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:47.387080 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:33:47.403600 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:47.403263 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" podStartSLOduration=1.403245717 podStartE2EDuration="1.403245717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:47.403127014 +0000 UTC m=+1068.038279126" watchObservedRunningTime="2026-09-15 21:33:47.403245717 +0000 UTC m=+1068.038397829" Sep 15 21:33:58.394626 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:33:58.394547 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:34:08.330350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.330320 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:08.330689 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.330372 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:08.522733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.522649 2552 generic.go:356] "Generic (PLEG): container finished" podID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerID="63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707" exitCode=0 Sep 15 21:34:08.522883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.522722 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerDied","Data":"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707"} Sep 15 21:34:08.523124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.523111 2552 scope.go:117] "RemoveContainer" containerID="63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707" Sep 15 21:34:08.524319 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.524291 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:08.524381 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.524358 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:08.524418 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:08.524388 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:09.531430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:09.531398 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerStarted","Data":"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366"} Sep 15 21:34:10.275485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:10.275400 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:10.275698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:10.275606 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:10.275698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:10.275675 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:11.275534 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:11.275500 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:11.275956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:11.275565 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:12.275250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:12.275212 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:12.275425 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:12.275281 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:13.275478 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:13.275447 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:13.275849 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:13.275502 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:14.274730 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:14.274692 2552 patch_prober.go:28] interesting pod/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 15 21:34:14.274951 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:14.274757 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 15 21:34:15.278840 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:15.278808 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:15.279247 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:15.279232 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:15.279977 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:15.279956 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:19.640857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.640810 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv"] Sep 15 21:34:19.643822 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.643808 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:19.646917 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.646896 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:34:19.654093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.654067 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv"] Sep 15 21:34:19.688380 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.688352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv\" (UID: \"74bbe358-d558-4102-8364-4ba007039598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:19.789023 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.788990 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv\" (UID: \"74bbe358-d558-4102-8364-4ba007039598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:19.789617 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.789590 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv\" (UID: \"74bbe358-d558-4102-8364-4ba007039598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:19.954812 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:19.954733 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:20.228869 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:20.228823 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv"] Sep 15 21:34:20.229883 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:34:20.229855 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bbe358_d558_4102_8364_4ba007039598.slice/crio-35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38 WatchSource:0}: Error finding container 35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38: Status 404 returned error can't find the container with id 35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38 Sep 15 21:34:20.613194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:20.613107 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" event={"ID":"74bbe358-d558-4102-8364-4ba007039598","Type":"ContainerStarted","Data":"0e5ce46352032aa7debe95e3534518cb023aa358d892ea83b3889dc7c6a7c51c"} Sep 15 21:34:20.613194 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:20.613173 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" event={"ID":"74bbe358-d558-4102-8364-4ba007039598","Type":"ContainerStarted","Data":"35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38"} Sep 15 21:34:20.631307 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:20.631260 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" podStartSLOduration=1.631245059 podStartE2EDuration="1.631245059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:20.628571576 +0000 UTC m=+1101.263723690" watchObservedRunningTime="2026-09-15 21:34:20.631245059 +0000 UTC m=+1101.266397169" Sep 15 21:34:21.620374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:21.620336 2552 generic.go:356] "Generic (PLEG): container finished" podID="74bbe358-d558-4102-8364-4ba007039598" containerID="0e5ce46352032aa7debe95e3534518cb023aa358d892ea83b3889dc7c6a7c51c" exitCode=0 Sep 15 21:34:21.620720 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:21.620409 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" event={"ID":"74bbe358-d558-4102-8364-4ba007039598","Type":"ContainerDied","Data":"0e5ce46352032aa7debe95e3534518cb023aa358d892ea83b3889dc7c6a7c51c"} Sep 15 21:34:22.762923 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:22.762900 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:22.816305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:22.816278 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca\") pod \"74bbe358-d558-4102-8364-4ba007039598\" (UID: \"74bbe358-d558-4102-8364-4ba007039598\") " Sep 15 21:34:22.816623 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:22.816603 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74bbe358-d558-4102-8364-4ba007039598" (UID: "74bbe358-d558-4102-8364-4ba007039598"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:34:22.816702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:22.816690 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bbe358-d558-4102-8364-4ba007039598-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:23.633818 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:23.633787 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" Sep 15 21:34:23.633991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:23.633846 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv" event={"ID":"74bbe358-d558-4102-8364-4ba007039598","Type":"ContainerDied","Data":"35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38"} Sep 15 21:34:23.633991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:23.633892 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35dcb8c7208c0928ad89b1f2c9f3d4469d64b34f6c7c0ee2a8b6ff3ee3935e38" Sep 15 21:34:30.253859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.253806 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:34:30.254664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.254616 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" podUID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" containerName="payload-pre-processing" containerID="cri-o://ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82" gracePeriod=30 Sep 15 21:34:30.450010 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.449395 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv"] Sep 15 21:34:30.456595 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.456568 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1664eb4-d4ggv"] Sep 15 21:34:30.501045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.501024 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:34:30.580252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.580156 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume\") pod \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " Sep 15 21:34:30.580252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.580216 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnwf8\" (UniqueName: \"kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8\") pod \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\" (UID: \"01af4df7-ecdf-4086-af11-3fd6c08e1a37\") " Sep 15 21:34:30.580588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.580556 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01af4df7-ecdf-4086-af11-3fd6c08e1a37" (UID: "01af4df7-ecdf-4086-af11-3fd6c08e1a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:34:30.582896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.582866 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8" (OuterVolumeSpecName: "kube-api-access-cnwf8") pod "01af4df7-ecdf-4086-af11-3fd6c08e1a37" (UID: "01af4df7-ecdf-4086-af11-3fd6c08e1a37"). InnerVolumeSpecName "kube-api-access-cnwf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:30.681098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.681067 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01af4df7-ecdf-4086-af11-3fd6c08e1a37-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:30.681098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.681094 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwf8\" (UniqueName: \"kubernetes.io/projected/01af4df7-ecdf-4086-af11-3fd6c08e1a37-kube-api-access-cnwf8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:30.683512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.683481 2552 generic.go:356] "Generic (PLEG): container finished" podID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" containerID="ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82" exitCode=0 Sep 15 21:34:30.683657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.683543 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" Sep 15 21:34:30.683657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.683564 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" event={"ID":"01af4df7-ecdf-4086-af11-3fd6c08e1a37","Type":"ContainerDied","Data":"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82"} Sep 15 21:34:30.683657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.683614 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7" event={"ID":"01af4df7-ecdf-4086-af11-3fd6c08e1a37","Type":"ContainerDied","Data":"5ac98f1562ba624c08a95c63a555237fb70afd5f5b3db5b327b4958eafa2368b"} Sep 15 21:34:30.683657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.683637 2552 scope.go:117] "RemoveContainer" containerID="ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82" Sep 15 21:34:30.693910 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.693892 2552 scope.go:117] "RemoveContainer" containerID="ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82" Sep 15 21:34:30.694220 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:34:30.694201 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82\": container with ID starting with ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82 not found: ID does not exist" containerID="ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82" Sep 15 21:34:30.694269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.694230 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82"} err="failed to get container status \"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82\": rpc error: code = NotFound desc = could not find container \"ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82\": container with ID starting with ea981ca74a5de7d77a38e88f22695104afd776bf7a296333a670e128e2ecaa82 not found: ID does not exist" Sep 15 21:34:30.706483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.706460 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:34:30.709747 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:30.709724 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1664eb4-74d5f695b6-74sk7"] Sep 15 21:34:31.796743 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:31.796709 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" path="/var/lib/kubelet/pods/01af4df7-ecdf-4086-af11-3fd6c08e1a37/volumes" Sep 15 21:34:31.797134 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:31.797060 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bbe358-d558-4102-8364-4ba007039598" path="/var/lib/kubelet/pods/74bbe358-d558-4102-8364-4ba007039598/volumes" Sep 15 21:34:37.113419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:37.113391 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:34:37.113954 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:37.113591 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" containerID="cri-o://2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366" gracePeriod=30 Sep 15 21:34:42.364160 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.364129 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:42.487186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487100 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487147 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487202 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlwsg\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487223 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487242 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487268 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487296 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487339 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487383 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data\") pod \"5e274bbb-d170-45e3-bfdc-e6629139b75f\" (UID: \"5e274bbb-d170-45e3-bfdc-e6629139b75f\") " Sep 15 21:34:42.487765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487513 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:42.487765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487694 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:34:42.487765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487761 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.487994 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487972 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:42.488118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487976 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:42.488118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.487985 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:42.489637 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.489614 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg" (OuterVolumeSpecName: "kube-api-access-mlwsg") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "kube-api-access-mlwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:42.489765 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.489742 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:42.489848 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.489759 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:34:42.489960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.489937 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e274bbb-d170-45e3-bfdc-e6629139b75f" (UID: "5e274bbb-d170-45e3-bfdc-e6629139b75f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:42.588442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588413 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588436 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588449 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588461 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588471 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwsg\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-kube-api-access-mlwsg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588484 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e274bbb-d170-45e3-bfdc-e6629139b75f-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588496 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e274bbb-d170-45e3-bfdc-e6629139b75f-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.588586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.588507 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e274bbb-d170-45e3-bfdc-e6629139b75f-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:34:42.761674 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.761605 2552 generic.go:356] "Generic (PLEG): container finished" podID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerID="2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366" exitCode=0 Sep 15 21:34:42.761674 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.761642 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerDied","Data":"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366"} Sep 15 21:34:42.761819 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.761696 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" Sep 15 21:34:42.761819 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.761741 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd" event={"ID":"5e274bbb-d170-45e3-bfdc-e6629139b75f","Type":"ContainerDied","Data":"d2fc7f9ea25a1973183531ad6605b5d4b499f3b3eb0948c67e5d5f64c913e09c"} Sep 15 21:34:42.761819 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.761767 2552 scope.go:117] "RemoveContainer" containerID="2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366" Sep 15 21:34:42.771870 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.771849 2552 scope.go:117] "RemoveContainer" containerID="63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707" Sep 15 21:34:42.780349 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.780316 2552 scope.go:117] "RemoveContainer" containerID="2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366" Sep 15 21:34:42.780741 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:34:42.780715 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366\": container with ID starting with 2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366 not found: ID does not exist" containerID="2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366" Sep 15 21:34:42.780866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.780754 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366"} err="failed to get container status \"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366\": rpc error: code = NotFound desc = could not find container \"2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366\": container with ID starting with 2165cf532db1aa4c5fe9e367dea83a47a4447054bdbb7e8169abeda5494ea366 not found: ID does not exist" Sep 15 21:34:42.780866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.780777 2552 scope.go:117] "RemoveContainer" containerID="63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707" Sep 15 21:34:42.781229 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:34:42.781209 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707\": container with ID starting with 63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707 not found: ID does not exist" containerID="63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707" Sep 15 21:34:42.781292 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.781233 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707"} err="failed to get container status \"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707\": rpc error: code = NotFound desc = could not find container \"63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707\": container with ID starting with 63bca16f7b8ee19605115c394b07e9ed01200389b56a7f76e325ce382cbf9707 not found: ID does not exist" Sep 15 21:34:42.783401 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.783382 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:34:42.787642 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:42.787624 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1664eb4-openshift-default-85c6f8488d-vzwpd"] Sep 15 21:34:43.790452 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:43.790421 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" path="/var/lib/kubelet/pods/5e274bbb-d170-45e3-bfdc-e6629139b75f/volumes" Sep 15 21:34:51.971291 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971255 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:34:51.971806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971790 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" containerName="payload-pre-processing" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971808 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" containerName="payload-pre-processing" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971839 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971848 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971874 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971882 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.971890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971889 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bbe358-d558-4102-8364-4ba007039598" containerName="revoke-keys" Sep 15 21:34:51.972074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971895 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bbe358-d558-4102-8364-4ba007039598" containerName="revoke-keys" Sep 15 21:34:51.972074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971965 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="01af4df7-ecdf-4086-af11-3fd6c08e1a37" containerName="payload-pre-processing" Sep 15 21:34:51.972074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971974 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.972074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971981 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e274bbb-d170-45e3-bfdc-e6629139b75f" containerName="istio-proxy" Sep 15 21:34:51.972074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.971987 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bbe358-d558-4102-8364-4ba007039598" containerName="revoke-keys" Sep 15 21:34:51.975147 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.975133 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:51.980155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:51.980132 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2d4fab-openshift-default-dockercfg-jjv4v\"" Sep 15 21:34:52.003471 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.003452 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:34:52.072438 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072389 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072470 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072523 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072558 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072810 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072666 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072810 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072717 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072810 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072750 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5pd\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072949 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072862 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.072949 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.072894 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.173942 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.173910 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.173957 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.173983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174011 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174210 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174303 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174273 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174402 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174309 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5pd\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174481 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174452 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174485 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174543 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174525 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174608 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174769 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.174848 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.174807 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.176588 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.176567 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.176690 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.176594 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.183065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.183038 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.183065 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.183055 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5pd\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd\") pod \"e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.285453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.285315 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:52.561923 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.561888 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:34:52.562974 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:34:52.562950 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf6c929_1e0b_4798_bfb4_d88ac2f78d0b.slice/crio-544e6341dc6102f5facadf1780e8ec43fe18c736fe3201ab0869410ad3484a80 WatchSource:0}: Error finding container 544e6341dc6102f5facadf1780e8ec43fe18c736fe3201ab0869410ad3484a80: Status 404 returned error can't find the container with id 544e6341dc6102f5facadf1780e8ec43fe18c736fe3201ab0869410ad3484a80 Sep 15 21:34:52.565255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.565220 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:52.565352 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.565287 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:52.565352 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.565325 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:34:52.836018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.835921 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerStarted","Data":"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d"} Sep 15 21:34:52.836018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.835971 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerStarted","Data":"544e6341dc6102f5facadf1780e8ec43fe18c736fe3201ab0869410ad3484a80"} Sep 15 21:34:52.860931 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:52.860874 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podStartSLOduration=1.8608559 podStartE2EDuration="1.8608559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:52.858808286 +0000 UTC m=+1133.493960398" watchObservedRunningTime="2026-09-15 21:34:52.8608559 +0000 UTC m=+1133.496008011" Sep 15 21:34:53.285631 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:53.285601 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:53.290422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:53.290402 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:53.842276 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:53.842246 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:34:53.843207 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:34:53.843183 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:00.634369 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.634332 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:35:00.638158 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.638138 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.640757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.640739 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2d4fab-dockercfg-nv6mc\"" Sep 15 21:35:00.641357 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.641341 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2d4fab\"" Sep 15 21:35:00.650385 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.650360 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:35:00.746350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.746316 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.746537 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.746365 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnj8\" (UniqueName: \"kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.847654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.847613 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.847823 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.847680 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnj8\" (UniqueName: \"kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.848300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.848281 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.857087 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.857065 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnj8\" (UniqueName: \"kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8\") pod \"payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:00.948567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:00.948507 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:01.221476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:01.221454 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:35:01.223474 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:35:01.223446 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ef56b3_fddf_41b9_b0ea_5d86c0b202bc.slice/crio-abebc1a75855d4428cd67055fbb640f3578d3d571a67323ba70a6d7018a581a8 WatchSource:0}: Error finding container abebc1a75855d4428cd67055fbb640f3578d3d571a67323ba70a6d7018a581a8: Status 404 returned error can't find the container with id abebc1a75855d4428cd67055fbb640f3578d3d571a67323ba70a6d7018a581a8 Sep 15 21:35:01.893266 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:01.893232 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" event={"ID":"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc","Type":"ContainerStarted","Data":"ff9ae42bc4c5b2d565c0f7dacf7226c0e1dd6d35d377eab4fdddaea2afa16949"} Sep 15 21:35:01.893651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:01.893281 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" event={"ID":"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc","Type":"ContainerStarted","Data":"abebc1a75855d4428cd67055fbb640f3578d3d571a67323ba70a6d7018a581a8"} Sep 15 21:35:01.893651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:01.893324 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:01.918317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:01.918271 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" podStartSLOduration=1.918258858 podStartE2EDuration="1.918258858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:01.915381292 +0000 UTC m=+1142.550533414" watchObservedRunningTime="2026-09-15 21:35:01.918258858 +0000 UTC m=+1142.553410969" Sep 15 21:35:12.901725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:12.901689 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:35:16.043491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.043455 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb"] Sep 15 21:35:16.046970 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.046947 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:16.051958 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.051933 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:35:16.057874 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.057779 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb"] Sep 15 21:35:16.076284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.076250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-32270d-tv7vb\" (UID: \"da190c2e-6348-4949-8f14-290914768854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:16.177610 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.177571 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-32270d-tv7vb\" (UID: \"da190c2e-6348-4949-8f14-290914768854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:16.178252 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.178233 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-32270d-tv7vb\" (UID: \"da190c2e-6348-4949-8f14-290914768854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:16.373646 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.373618 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:16.661248 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.648802 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb"] Sep 15 21:35:16.982219 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.982190 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" event={"ID":"da190c2e-6348-4949-8f14-290914768854","Type":"ContainerStarted","Data":"b3bf049df8405f779d3d22608338bcf4f7ebb2053b4861680cbea96a1c0bb7ec"} Sep 15 21:35:16.982341 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:16.982242 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" event={"ID":"da190c2e-6348-4949-8f14-290914768854","Type":"ContainerStarted","Data":"27d9e9eb2753651476ed329ece73be47cb8facb617af01291d588dd043d60035"} Sep 15 21:35:17.002817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:17.002774 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" podStartSLOduration=1.002758564 podStartE2EDuration="1.002758564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:17.001921932 +0000 UTC m=+1157.637074045" watchObservedRunningTime="2026-09-15 21:35:17.002758564 +0000 UTC m=+1157.637910674" Sep 15 21:35:17.990047 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:17.989957 2552 generic.go:356] "Generic (PLEG): container finished" podID="da190c2e-6348-4949-8f14-290914768854" containerID="b3bf049df8405f779d3d22608338bcf4f7ebb2053b4861680cbea96a1c0bb7ec" exitCode=0 Sep 15 21:35:17.990401 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:17.990040 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" event={"ID":"da190c2e-6348-4949-8f14-290914768854","Type":"ContainerDied","Data":"b3bf049df8405f779d3d22608338bcf4f7ebb2053b4861680cbea96a1c0bb7ec"} Sep 15 21:35:19.172555 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:19.172530 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:19.208052 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:19.208026 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca\") pod \"da190c2e-6348-4949-8f14-290914768854\" (UID: \"da190c2e-6348-4949-8f14-290914768854\") " Sep 15 21:35:19.208370 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:19.208346 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da190c2e-6348-4949-8f14-290914768854" (UID: "da190c2e-6348-4949-8f14-290914768854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:35:19.309458 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:19.309382 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da190c2e-6348-4949-8f14-290914768854-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:35:20.007721 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:20.007634 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" Sep 15 21:35:20.007934 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:20.007631 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb" event={"ID":"da190c2e-6348-4949-8f14-290914768854","Type":"ContainerDied","Data":"27d9e9eb2753651476ed329ece73be47cb8facb617af01291d588dd043d60035"} Sep 15 21:35:20.007934 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:20.007911 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d9e9eb2753651476ed329ece73be47cb8facb617af01291d588dd043d60035" Sep 15 21:35:23.029581 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.029512 2552 generic.go:356] "Generic (PLEG): container finished" podID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerID="783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d" exitCode=0 Sep 15 21:35:23.029980 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.029584 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerDied","Data":"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d"} Sep 15 21:35:23.030071 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.030051 2552 scope.go:117] "RemoveContainer" containerID="783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d" Sep 15 21:35:23.031171 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.031140 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:23.031279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.031209 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:23.031279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.031258 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:23.842578 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:23.842542 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:24.037888 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:24.037850 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerStarted","Data":"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330"} Sep 15 21:35:24.285904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:24.285871 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:24.287204 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:24.287177 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 15 21:35:24.287346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:24.287227 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 15 21:35:25.286022 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.285987 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 15 21:35:25.286402 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.286037 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 15 21:35:25.768879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.768268 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:35:25.769124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.769100 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da190c2e-6348-4949-8f14-290914768854" containerName="revoke-keys" Sep 15 21:35:25.769124 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.769125 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="da190c2e-6348-4949-8f14-290914768854" containerName="revoke-keys" Sep 15 21:35:25.769312 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.769276 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="da190c2e-6348-4949-8f14-290914768854" containerName="revoke-keys" Sep 15 21:35:25.773639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.772588 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.776181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.776155 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:35:25.776955 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.776646 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fb0f0e-openshift-default-dockercfg-bjhcq\"" Sep 15 21:35:25.877982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.877946 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.877982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.877985 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878012 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqn5\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878038 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878105 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878132 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878171 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.878450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.878291 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.978809 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.978881 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.978928 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.978997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.979053 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.979124 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.979172 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.979199 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqn5\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.979225 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.980148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.980273 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.981186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.980964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.981302 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.981274 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.981483 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.981445 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.981889 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.981862 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.983643 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.983375 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.985221 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.985159 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.990921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.990006 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqn5\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:25.990921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:25.990428 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token\") pod \"e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:26.086138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.086061 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:26.289186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.289147 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 15 21:35:26.289569 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.289206 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 15 21:35:26.384855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.384808 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:35:26.385269 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:35:26.385243 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726a252e_90d8_4518_9024_d0062c2b242e.slice/crio-dfd81275467a8429ad8b42101e1f93dce6c0a756a5bbda1a566f73b35d4be33c WatchSource:0}: Error finding container dfd81275467a8429ad8b42101e1f93dce6c0a756a5bbda1a566f73b35d4be33c: Status 404 returned error can't find the container with id dfd81275467a8429ad8b42101e1f93dce6c0a756a5bbda1a566f73b35d4be33c Sep 15 21:35:26.387609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.387576 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:26.387702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.387651 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:26.387702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:26.387694 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:35:27.067903 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.067857 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerStarted","Data":"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd"} Sep 15 21:35:27.068098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.067919 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerStarted","Data":"dfd81275467a8429ad8b42101e1f93dce6c0a756a5bbda1a566f73b35d4be33c"} Sep 15 21:35:27.086936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.086907 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:27.090078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.089980 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podStartSLOduration=2.089949469 podStartE2EDuration="2.089949469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:27.088500947 +0000 UTC m=+1167.723653081" watchObservedRunningTime="2026-09-15 21:35:27.089949469 +0000 UTC m=+1167.725101581" Sep 15 21:35:27.093773 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.093751 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:27.286166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.286131 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 15 21:35:27.286333 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.286184 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 15 21:35:27.761573 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.761529 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb"] Sep 15 21:35:27.770659 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.770634 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-32270d-tv7vb"] Sep 15 21:35:27.790922 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:27.790894 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da190c2e-6348-4949-8f14-290914768854" path="/var/lib/kubelet/pods/da190c2e-6348-4949-8f14-290914768854/volumes" Sep 15 21:35:28.075210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:28.075137 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:28.076118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:28.076092 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:35:28.286701 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:28.286192 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 15 21:35:28.286701 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:28.286274 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 15 21:35:29.373705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:29.373660 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:29.374215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:29.374108 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:29.376450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:29.376427 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:35:59.835665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:59.835634 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:35:59.867284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:59.867260 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:35:59.962878 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:35:59.962846 2552 scope.go:117] "RemoveContainer" containerID="904ccc8bc97965232bd28c58249cc746c5ad2f80171797f23eead70887bd80a9" Sep 15 21:36:04.318491 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.318457 2552 generic.go:356] "Generic (PLEG): container finished" podID="726a252e-90d8-4518-9024-d0062c2b242e" containerID="0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd" exitCode=0 Sep 15 21:36:04.318905 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.318519 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerDied","Data":"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd"} Sep 15 21:36:04.318961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.318914 2552 scope.go:117] "RemoveContainer" containerID="0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd" Sep 15 21:36:04.319967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.319939 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:36:04.320046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.320010 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:36:04.320081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:04.320043 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:36:05.327706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:05.327673 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerStarted","Data":"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7"} Sep 15 21:36:06.087052 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:06.087015 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:36:06.087305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:06.087283 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:36:06.087366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:06.087331 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 15 21:36:07.087253 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:07.087219 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:36:07.087650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:07.087298 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 15 21:36:08.086797 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:08.086760 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:36:08.087016 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:08.086814 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 15 21:36:09.086430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:09.086393 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:36:09.086815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:09.086450 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 15 21:36:10.086427 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:10.086391 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 15 21:36:10.086607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:10.086459 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 15 21:36:11.090238 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:11.090207 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:36:11.090683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:11.090662 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:36:11.091397 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:11.091381 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:36:17.565599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.565563 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:36:17.568241 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.568223 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.572313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.572295 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-56820a3c\"" Sep 15 21:36:17.572460 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.572426 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-56820a3c-dockercfg-mr2rs\"" Sep 15 21:36:17.593587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.593565 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:36:17.653556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.653519 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.653774 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.653748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwczt\" (UniqueName: \"kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.754883 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.754819 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwczt\" (UniqueName: \"kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.755027 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.754941 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.755621 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.755601 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.764867 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.764822 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwczt\" (UniqueName: \"kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt\") pod \"payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:17.878967 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:17.878939 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:18.157575 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:18.157546 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:36:18.158457 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:36:18.158427 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bbe11e_e931_400f_815d_6872cead0df2.slice/crio-d4572b5d2f5580f5db06f7a79d4c8bfe77e655c0709ddd324729e014f4a2269f WatchSource:0}: Error finding container d4572b5d2f5580f5db06f7a79d4c8bfe77e655c0709ddd324729e014f4a2269f: Status 404 returned error can't find the container with id d4572b5d2f5580f5db06f7a79d4c8bfe77e655c0709ddd324729e014f4a2269f Sep 15 21:36:18.429515 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:18.429436 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" event={"ID":"77bbe11e-e931-400f-815d-6872cead0df2","Type":"ContainerStarted","Data":"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64"} Sep 15 21:36:18.429515 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:18.429488 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" event={"ID":"77bbe11e-e931-400f-815d-6872cead0df2","Type":"ContainerStarted","Data":"d4572b5d2f5580f5db06f7a79d4c8bfe77e655c0709ddd324729e014f4a2269f"} Sep 15 21:36:18.429701 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:18.429556 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:18.447980 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:18.447938 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" podStartSLOduration=1.447926606 podStartE2EDuration="1.447926606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:18.446822168 +0000 UTC m=+1219.081974291" watchObservedRunningTime="2026-09-15 21:36:18.447926606 +0000 UTC m=+1219.083078741" Sep 15 21:36:22.877271 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:22.877238 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf"] Sep 15 21:36:22.879816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:22.879799 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:22.882567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:22.882536 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:36:22.889669 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:22.889646 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf"] Sep 15 21:36:23.003042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:23.003006 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf\" (UID: \"dc150cf5-66ee-4bf4-8ca7-d34402719495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:23.103611 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:23.103579 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf\" (UID: \"dc150cf5-66ee-4bf4-8ca7-d34402719495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:23.104182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:23.104163 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf\" (UID: \"dc150cf5-66ee-4bf4-8ca7-d34402719495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:23.191813 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:23.191746 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:23.466736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:23.466709 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf"] Sep 15 21:36:23.468002 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:36:23.467981 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc150cf5_66ee_4bf4_8ca7_d34402719495.slice/crio-958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906 WatchSource:0}: Error finding container 958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906: Status 404 returned error can't find the container with id 958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906 Sep 15 21:36:24.469369 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:24.469334 2552 generic.go:356] "Generic (PLEG): container finished" podID="dc150cf5-66ee-4bf4-8ca7-d34402719495" containerID="348bd5855aed0c9376bffaeb44f33a54a56b972257cbf384fd7bed9a6219ca29" exitCode=0 Sep 15 21:36:24.469705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:24.469412 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" event={"ID":"dc150cf5-66ee-4bf4-8ca7-d34402719495","Type":"ContainerDied","Data":"348bd5855aed0c9376bffaeb44f33a54a56b972257cbf384fd7bed9a6219ca29"} Sep 15 21:36:24.469705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:24.469464 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" event={"ID":"dc150cf5-66ee-4bf4-8ca7-d34402719495","Type":"ContainerStarted","Data":"958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906"} Sep 15 21:36:25.623741 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:25.623718 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:25.725925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:25.725894 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca\") pod \"dc150cf5-66ee-4bf4-8ca7-d34402719495\" (UID: \"dc150cf5-66ee-4bf4-8ca7-d34402719495\") " Sep 15 21:36:25.726231 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:25.726209 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc150cf5-66ee-4bf4-8ca7-d34402719495" (UID: "dc150cf5-66ee-4bf4-8ca7-d34402719495"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:25.827424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:25.827396 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc150cf5-66ee-4bf4-8ca7-d34402719495-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:26.483462 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:26.483437 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" Sep 15 21:36:26.483650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:26.483478 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf" event={"ID":"dc150cf5-66ee-4bf4-8ca7-d34402719495","Type":"ContainerDied","Data":"958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906"} Sep 15 21:36:26.483650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:26.483552 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="958d5cbdce1eb4620510b189a669ac897580124b7a197e839e735b063d807906" Sep 15 21:36:27.178498 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:27.178464 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:36:27.178969 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:27.178754 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" containerID="cri-o://2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2" gracePeriod=30 Sep 15 21:36:28.418641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:28.418605 2552 patch_prober.go:28] interesting pod/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:36:28.419120 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:28.418673 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:36:29.438029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:29.438001 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:36:32.441847 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.441778 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:36:32.526245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.526218 2552 generic.go:356] "Generic (PLEG): container finished" podID="b57d90e9-9334-42eb-98b3-083150ed691c" containerID="2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2" exitCode=0 Sep 15 21:36:32.526394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.526263 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" Sep 15 21:36:32.526394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.526304 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerDied","Data":"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2"} Sep 15 21:36:32.526512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.526392 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94" event={"ID":"b57d90e9-9334-42eb-98b3-083150ed691c","Type":"ContainerDied","Data":"9656f249db57a1b17aabfa103dd9b25f040b61db1d960c46f667f9db5e66a1e9"} Sep 15 21:36:32.526512 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.526424 2552 scope.go:117] "RemoveContainer" containerID="2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2" Sep 15 21:36:32.536021 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.536003 2552 scope.go:117] "RemoveContainer" containerID="aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928" Sep 15 21:36:32.544519 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.544503 2552 scope.go:117] "RemoveContainer" containerID="2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2" Sep 15 21:36:32.544782 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:36:32.544757 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2\": container with ID starting with 2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2 not found: ID does not exist" containerID="2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2" Sep 15 21:36:32.544915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.544790 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2"} err="failed to get container status \"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2\": rpc error: code = NotFound desc = could not find container \"2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2\": container with ID starting with 2c7a7be69bb8223b0f829d4f6746eac2c9ab432100010fc2303e663a00a14cf2 not found: ID does not exist" Sep 15 21:36:32.544915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.544808 2552 scope.go:117] "RemoveContainer" containerID="aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928" Sep 15 21:36:32.545101 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:36:32.545085 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928\": container with ID starting with aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928 not found: ID does not exist" containerID="aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928" Sep 15 21:36:32.545149 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.545109 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928"} err="failed to get container status \"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928\": rpc error: code = NotFound desc = could not find container \"aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928\": container with ID starting with aa02645be886d0b387de5eda8caadaf5baeede97ae0af8a8a99274fc4f1b9928 not found: ID does not exist" Sep 15 21:36:32.596354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596334 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596369 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596406 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596421 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596463 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596451 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596515 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596550 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596583 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4vpw\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596594 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596622 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy\") pod \"b57d90e9-9334-42eb-98b3-083150ed691c\" (UID: \"b57d90e9-9334-42eb-98b3-083150ed691c\") " Sep 15 21:36:32.596686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596642 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596778 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596853 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596882 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596974 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57d90e9-9334-42eb-98b3-083150ed691c-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.596993 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.597012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.597007 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.597310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.597020 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.597310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.597033 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.598580 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.598556 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:32.598677 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.598651 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw" (OuterVolumeSpecName: "kube-api-access-k4vpw") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "kube-api-access-k4vpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:32.598753 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.598739 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:32.598799 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.598784 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b57d90e9-9334-42eb-98b3-083150ed691c" (UID: "b57d90e9-9334-42eb-98b3-083150ed691c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:36:32.697666 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.697613 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.697666 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.697634 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57d90e9-9334-42eb-98b3-083150ed691c-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.697666 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.697643 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vpw\" (UniqueName: \"kubernetes.io/projected/b57d90e9-9334-42eb-98b3-083150ed691c-kube-api-access-k4vpw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.697666 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.697653 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57d90e9-9334-42eb-98b3-083150ed691c-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:32.850417 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.850387 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:36:32.855383 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:32.855360 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-32270d-openshift-default-798c96bdcb-95m94"] Sep 15 21:36:33.790540 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:33.790504 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" path="/var/lib/kubelet/pods/b57d90e9-9334-42eb-98b3-083150ed691c/volumes" Sep 15 21:36:36.003987 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.003954 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9"] Sep 15 21:36:36.006686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.006662 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4bc420-29825130-mbbp9"] Sep 15 21:36:36.028125 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.028094 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:36:36.028385 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.028335 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" podUID="dabf3e48-4951-48ac-ad52-b76cf422ea85" containerName="payload-processing" containerID="cri-o://e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af" gracePeriod=30 Sep 15 21:36:36.210974 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.210941 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf"] Sep 15 21:36:36.213891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.213866 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4bc420-w74rf"] Sep 15 21:36:36.276866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.276814 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:36:36.432472 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.432438 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume\") pod \"dabf3e48-4951-48ac-ad52-b76cf422ea85\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " Sep 15 21:36:36.432620 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.432527 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tt9s\" (UniqueName: \"kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s\") pod \"dabf3e48-4951-48ac-ad52-b76cf422ea85\" (UID: \"dabf3e48-4951-48ac-ad52-b76cf422ea85\") " Sep 15 21:36:36.432888 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.432865 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dabf3e48-4951-48ac-ad52-b76cf422ea85" (UID: "dabf3e48-4951-48ac-ad52-b76cf422ea85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:36.434519 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.434496 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s" (OuterVolumeSpecName: "kube-api-access-6tt9s") pod "dabf3e48-4951-48ac-ad52-b76cf422ea85" (UID: "dabf3e48-4951-48ac-ad52-b76cf422ea85"). InnerVolumeSpecName "kube-api-access-6tt9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:36.534140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.534072 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tt9s\" (UniqueName: \"kubernetes.io/projected/dabf3e48-4951-48ac-ad52-b76cf422ea85-kube-api-access-6tt9s\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:36.534140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.534099 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabf3e48-4951-48ac-ad52-b76cf422ea85-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:36.551305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.551271 2552 generic.go:356] "Generic (PLEG): container finished" podID="dabf3e48-4951-48ac-ad52-b76cf422ea85" containerID="e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af" exitCode=0 Sep 15 21:36:36.551423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.551344 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" Sep 15 21:36:36.551423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.551356 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" event={"ID":"dabf3e48-4951-48ac-ad52-b76cf422ea85","Type":"ContainerDied","Data":"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af"} Sep 15 21:36:36.551513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.551432 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp" event={"ID":"dabf3e48-4951-48ac-ad52-b76cf422ea85","Type":"ContainerDied","Data":"3d0beb2b9bbf4f30f08c8594b8c920ead62363179cc1844a791deb795d17810a"} Sep 15 21:36:36.551513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.551453 2552 scope.go:117] "RemoveContainer" containerID="e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af" Sep 15 21:36:36.560859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.560809 2552 scope.go:117] "RemoveContainer" containerID="e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af" Sep 15 21:36:36.561092 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:36:36.561074 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af\": container with ID starting with e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af not found: ID does not exist" containerID="e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af" Sep 15 21:36:36.561143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.561100 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af"} err="failed to get container status \"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af\": rpc error: code = NotFound desc = could not find container \"e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af\": container with ID starting with e52a85960d47d83d79d7d6bf3e40ed9ea61d25a3352940af81125f456392f7af not found: ID does not exist" Sep 15 21:36:36.573089 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.573063 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:36:36.578479 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:36.578458 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4bc420-748bffd5b7-65mnp"] Sep 15 21:36:37.793850 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:37.793748 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70e64bf-9f9e-44bd-a41b-42b5286e1117" path="/var/lib/kubelet/pods/c70e64bf-9f9e-44bd-a41b-42b5286e1117/volumes" Sep 15 21:36:37.794197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:37.794128 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabf3e48-4951-48ac-ad52-b76cf422ea85" path="/var/lib/kubelet/pods/dabf3e48-4951-48ac-ad52-b76cf422ea85/volumes" Sep 15 21:36:37.794413 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:37.794400 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc150cf5-66ee-4bf4-8ca7-d34402719495" path="/var/lib/kubelet/pods/dc150cf5-66ee-4bf4-8ca7-d34402719495/volumes" Sep 15 21:36:43.279887 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.279852 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5"] Sep 15 21:36:43.280292 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280278 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280292 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280291 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280302 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabf3e48-4951-48ac-ad52-b76cf422ea85" containerName="payload-processing" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280308 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabf3e48-4951-48ac-ad52-b76cf422ea85" containerName="payload-processing" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280315 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280320 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280331 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc150cf5-66ee-4bf4-8ca7-d34402719495" containerName="revoke-keys" Sep 15 21:36:43.280377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280337 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc150cf5-66ee-4bf4-8ca7-d34402719495" containerName="revoke-keys" Sep 15 21:36:43.280583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280412 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280419 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabf3e48-4951-48ac-ad52-b76cf422ea85" containerName="payload-processing" Sep 15 21:36:43.280583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280425 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57d90e9-9334-42eb-98b3-083150ed691c" containerName="istio-proxy" Sep 15 21:36:43.280583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.280431 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc150cf5-66ee-4bf4-8ca7-d34402719495" containerName="revoke-keys" Sep 15 21:36:43.283596 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.283578 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:43.285981 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.285961 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:36:43.293411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.293387 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5"] Sep 15 21:36:43.392563 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.392534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5\" (UID: \"33b8aec0-86da-4d30-882c-ea9d723071c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:43.493488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.493462 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5\" (UID: \"33b8aec0-86da-4d30-882c-ea9d723071c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:43.494082 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.494065 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5\" (UID: \"33b8aec0-86da-4d30-882c-ea9d723071c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:43.594876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.594798 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:43.868966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:43.868938 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5"] Sep 15 21:36:43.871694 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:36:43.871665 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b8aec0_86da_4d30_882c_ea9d723071c9.slice/crio-8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434 WatchSource:0}: Error finding container 8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434: Status 404 returned error can't find the container with id 8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434 Sep 15 21:36:44.604093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:44.604067 2552 generic.go:356] "Generic (PLEG): container finished" podID="33b8aec0-86da-4d30-882c-ea9d723071c9" containerID="97656c9df9a6b7701dd481f8b627f500a0315e147e27ba35160c4a981cad7a0b" exitCode=0 Sep 15 21:36:44.604422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:44.604159 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" event={"ID":"33b8aec0-86da-4d30-882c-ea9d723071c9","Type":"ContainerDied","Data":"97656c9df9a6b7701dd481f8b627f500a0315e147e27ba35160c4a981cad7a0b"} Sep 15 21:36:44.604422 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:44.604229 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" event={"ID":"33b8aec0-86da-4d30-882c-ea9d723071c9","Type":"ContainerStarted","Data":"8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434"} Sep 15 21:36:45.750982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:45.750957 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:45.916102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:45.916074 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca\") pod \"33b8aec0-86da-4d30-882c-ea9d723071c9\" (UID: \"33b8aec0-86da-4d30-882c-ea9d723071c9\") " Sep 15 21:36:45.916468 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:45.916445 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33b8aec0-86da-4d30-882c-ea9d723071c9" (UID: "33b8aec0-86da-4d30-882c-ea9d723071c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:46.017576 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:46.017540 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b8aec0-86da-4d30-882c-ea9d723071c9-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:46.618889 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:46.618858 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" event={"ID":"33b8aec0-86da-4d30-882c-ea9d723071c9","Type":"ContainerDied","Data":"8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434"} Sep 15 21:36:46.618889 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:46.618877 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5" Sep 15 21:36:46.619098 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:46.618905 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b847a6cfc4e6b81b576c0368d91e1a1b031aa7a4b670e44aac4f06b938ba434" Sep 15 21:36:53.996991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:53.996917 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg"] Sep 15 21:36:54.000132 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.000108 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bb7867-29825130-f7gdg"] Sep 15 21:36:54.027991 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.027965 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:36:54.028191 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.028168 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" podUID="575bcf76-84b1-446d-9282-a71bc56ee3dc" containerName="payload-processing" containerID="cri-o://7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c" gracePeriod=30 Sep 15 21:36:54.246278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.246242 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5"] Sep 15 21:36:54.249936 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.249866 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb7867-mm5t5"] Sep 15 21:36:54.297628 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.297610 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:36:54.387963 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.387936 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lw69\" (UniqueName: \"kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69\") pod \"575bcf76-84b1-446d-9282-a71bc56ee3dc\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " Sep 15 21:36:54.388141 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.387985 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume\") pod \"575bcf76-84b1-446d-9282-a71bc56ee3dc\" (UID: \"575bcf76-84b1-446d-9282-a71bc56ee3dc\") " Sep 15 21:36:54.388385 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.388360 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "575bcf76-84b1-446d-9282-a71bc56ee3dc" (UID: "575bcf76-84b1-446d-9282-a71bc56ee3dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:54.390010 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.389987 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69" (OuterVolumeSpecName: "kube-api-access-6lw69") pod "575bcf76-84b1-446d-9282-a71bc56ee3dc" (UID: "575bcf76-84b1-446d-9282-a71bc56ee3dc"). InnerVolumeSpecName "kube-api-access-6lw69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:54.489480 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.489451 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lw69\" (UniqueName: \"kubernetes.io/projected/575bcf76-84b1-446d-9282-a71bc56ee3dc-kube-api-access-6lw69\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.489599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.489488 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575bcf76-84b1-446d-9282-a71bc56ee3dc-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.668663 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.668632 2552 generic.go:356] "Generic (PLEG): container finished" podID="575bcf76-84b1-446d-9282-a71bc56ee3dc" containerID="7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c" exitCode=0 Sep 15 21:36:54.668806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.668692 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" Sep 15 21:36:54.668806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.668691 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" event={"ID":"575bcf76-84b1-446d-9282-a71bc56ee3dc","Type":"ContainerDied","Data":"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c"} Sep 15 21:36:54.668938 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.668807 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk" event={"ID":"575bcf76-84b1-446d-9282-a71bc56ee3dc","Type":"ContainerDied","Data":"addad44678b7dd1ee5f41e31c758671452e37230f256573d1c93dcd20b401638"} Sep 15 21:36:54.668938 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.668864 2552 scope.go:117] "RemoveContainer" containerID="7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c" Sep 15 21:36:54.678256 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.678241 2552 scope.go:117] "RemoveContainer" containerID="7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c" Sep 15 21:36:54.678540 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:36:54.678522 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c\": container with ID starting with 7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c not found: ID does not exist" containerID="7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c" Sep 15 21:36:54.678580 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.678550 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c"} err="failed to get container status \"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c\": rpc error: code = NotFound desc = could not find container \"7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c\": container with ID starting with 7a51919304325941c092402567c7b840b31ab7d4a7e5381dd852edda8656122c not found: ID does not exist" Sep 15 21:36:54.692311 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.692290 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:36:54.696715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:54.696695 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bb7867-579cb9d775-bfptk"] Sep 15 21:36:55.790465 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:55.790433 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b8aec0-86da-4d30-882c-ea9d723071c9" path="/var/lib/kubelet/pods/33b8aec0-86da-4d30-882c-ea9d723071c9/volumes" Sep 15 21:36:55.790820 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:55.790790 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575bcf76-84b1-446d-9282-a71bc56ee3dc" path="/var/lib/kubelet/pods/575bcf76-84b1-446d-9282-a71bc56ee3dc/volumes" Sep 15 21:36:55.791081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:55.791068 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9df8ace-7812-49b3-b40c-fc46cb10ed82" path="/var/lib/kubelet/pods/c9df8ace-7812-49b3-b40c-fc46cb10ed82/volumes" Sep 15 21:36:59.986688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:59.986659 2552 scope.go:117] "RemoveContainer" containerID="06e9f266093067c7202a2b6a7eac8ffc570b829bb35b698930e1bd04b980b75d" Sep 15 21:36:59.998821 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:36:59.998798 2552 scope.go:117] "RemoveContainer" containerID="b6b32130d91968a972fb67b42b39e12b2cfc26ad3179c4b33e23996f085714c7" Sep 15 21:37:00.012114 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:00.012094 2552 scope.go:117] "RemoveContainer" containerID="ee52a6e10e7a7b9add38220010ce5da8b2da15730bd8469e7405b61b02ff6b7e" Sep 15 21:37:00.026514 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:00.026242 2552 scope.go:117] "RemoveContainer" containerID="4f6f64c6d83ba5e5adc7ebe3d3eb566b376c555c87296f0c7e312e1493f5b236" Sep 15 21:37:14.447616 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.447580 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:37:14.448261 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448239 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575bcf76-84b1-446d-9282-a71bc56ee3dc" containerName="payload-processing" Sep 15 21:37:14.448346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448265 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="575bcf76-84b1-446d-9282-a71bc56ee3dc" containerName="payload-processing" Sep 15 21:37:14.448346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448278 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b8aec0-86da-4d30-882c-ea9d723071c9" containerName="revoke-keys" Sep 15 21:37:14.448346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448286 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b8aec0-86da-4d30-882c-ea9d723071c9" containerName="revoke-keys" Sep 15 21:37:14.448556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448435 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="575bcf76-84b1-446d-9282-a71bc56ee3dc" containerName="payload-processing" Sep 15 21:37:14.448556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.448450 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b8aec0-86da-4d30-882c-ea9d723071c9" containerName="revoke-keys" Sep 15 21:37:14.452891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.452867 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.457117 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.457091 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-59e243-openshift-default-dockercfg-qbhll\"" Sep 15 21:37:14.478390 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.478362 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:37:14.578364 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.578323 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.578615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.578595 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.578755 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.578739 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.578897 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.578882 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.579037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.579024 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.579137 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.579124 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.579247 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.579234 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.579348 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.579334 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4zz\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.579467 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.579454 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681130 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681192 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681247 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681301 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681329 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681372 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681400 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4zz\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.681925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.681461 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.682999 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.682973 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.683509 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.683484 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.684478 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.684149 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.684647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.684298 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.684780 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.684410 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.687200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.687128 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.692845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.687629 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.696999 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.696380 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.697351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.697328 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4zz\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz\") pod \"e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:14.778482 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:14.778403 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:15.075590 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.075564 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:37:15.077225 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:37:15.077198 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b89ff44_b881_4ad3_8fc9_f52026cd8bd5.slice/crio-4636507e16d24a32c796e00405e459c6cb324657454bf0cf57f10abc1a556811 WatchSource:0}: Error finding container 4636507e16d24a32c796e00405e459c6cb324657454bf0cf57f10abc1a556811: Status 404 returned error can't find the container with id 4636507e16d24a32c796e00405e459c6cb324657454bf0cf57f10abc1a556811 Sep 15 21:37:15.079715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.079676 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:15.079782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.079766 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:15.079953 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.079932 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:15.811607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.811572 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerStarted","Data":"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139"} Sep 15 21:37:15.811997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.811629 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerStarted","Data":"4636507e16d24a32c796e00405e459c6cb324657454bf0cf57f10abc1a556811"} Sep 15 21:37:15.833870 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:15.833802 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podStartSLOduration=1.8337848939999999 podStartE2EDuration="1.833784894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:15.831045184 +0000 UTC m=+1276.466197287" watchObservedRunningTime="2026-09-15 21:37:15.833784894 +0000 UTC m=+1276.468937006" Sep 15 21:37:16.779202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:16.779175 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:16.784354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:16.784332 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:16.818018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:16.817986 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:16.818757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:16.818740 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:23.308415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.308378 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:37:23.312237 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.312218 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.315015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.314995 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-59e243\"" Sep 15 21:37:23.315298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.315282 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-59e243-dockercfg-wcfqp\"" Sep 15 21:37:23.327612 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.327588 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:37:23.471017 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.470985 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.471186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.471046 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq59p\" (UniqueName: \"kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.571592 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.571502 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.571592 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.571566 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq59p\" (UniqueName: \"kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.572200 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.572177 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.580391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.580368 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq59p\" (UniqueName: \"kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p\") pod \"payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.622835 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.622812 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:23.896127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:23.896097 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:37:23.897814 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:37:23.897794 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1deb4f5e_7a96_4156_9fa1_50b34415ab77.slice/crio-68815204f8a756dd18df0f3d3ed2ebef44279f5b06d6fe2a1bb7fc07212a32e1 WatchSource:0}: Error finding container 68815204f8a756dd18df0f3d3ed2ebef44279f5b06d6fe2a1bb7fc07212a32e1: Status 404 returned error can't find the container with id 68815204f8a756dd18df0f3d3ed2ebef44279f5b06d6fe2a1bb7fc07212a32e1 Sep 15 21:37:24.869814 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:24.869778 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" event={"ID":"1deb4f5e-7a96-4156-9fa1-50b34415ab77","Type":"ContainerStarted","Data":"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69"} Sep 15 21:37:24.870186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:24.869847 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" event={"ID":"1deb4f5e-7a96-4156-9fa1-50b34415ab77","Type":"ContainerStarted","Data":"68815204f8a756dd18df0f3d3ed2ebef44279f5b06d6fe2a1bb7fc07212a32e1"} Sep 15 21:37:24.870186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:24.870007 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:24.887468 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:24.887422 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" podStartSLOduration=1.887408281 podStartE2EDuration="1.887408281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:24.884851959 +0000 UTC m=+1285.520004071" watchObservedRunningTime="2026-09-15 21:37:24.887408281 +0000 UTC m=+1285.522560423" Sep 15 21:37:35.876657 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:35.876623 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:37:47.859989 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:47.859950 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 15 21:37:47.860409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:47.860016 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 15 21:37:48.417382 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.417352 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:37:48.422518 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.422488 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.425700 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.425678 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-38a3a3-openshift-default-dockercfg-8nfp9\"" Sep 15 21:37:48.450506 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.450483 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:37:48.511908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.511872 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512191 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512164 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512237 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512276 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzzt\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512324 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512521 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512452 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512596 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512546 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512589 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.512651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.512616 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.613562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613523 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.613562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613564 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.613815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613603 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.613815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613629 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzzt\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.613815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613745 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613855 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613939 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.613970 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614228 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.614084 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614527 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.614331 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.614670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.614601 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.615119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.615088 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.615739 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.615713 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.616968 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.616939 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.617556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.617535 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.630678 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.630652 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.631510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.631481 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzzt\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt\") pod \"e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:48.733169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:48.733088 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:49.016142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.016111 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:37:49.017365 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:37:49.017336 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482fb2c_5c54_49e2_9861_fb3992fed332.slice/crio-5a4f390d8bc6403344c0fa052b7d782202ad5b56ea793869b4a3138ddd49e782 WatchSource:0}: Error finding container 5a4f390d8bc6403344c0fa052b7d782202ad5b56ea793869b4a3138ddd49e782: Status 404 returned error can't find the container with id 5a4f390d8bc6403344c0fa052b7d782202ad5b56ea793869b4a3138ddd49e782 Sep 15 21:37:49.019643 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.019610 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:49.019733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.019678 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:49.019733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.019708 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:49.024584 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.024560 2552 generic.go:356] "Generic (PLEG): container finished" podID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerID="018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139" exitCode=0 Sep 15 21:37:49.024671 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.024614 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerDied","Data":"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139"} Sep 15 21:37:49.025008 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.024994 2552 scope.go:117] "RemoveContainer" containerID="018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139" Sep 15 21:37:49.025990 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.025961 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:49.026090 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.026029 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:49.026090 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:49.026059 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:37:50.033350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.033314 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerStarted","Data":"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa"} Sep 15 21:37:50.035018 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.034993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerStarted","Data":"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c"} Sep 15 21:37:50.035133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.035034 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerStarted","Data":"5a4f390d8bc6403344c0fa052b7d782202ad5b56ea793869b4a3138ddd49e782"} Sep 15 21:37:50.081076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.081010 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podStartSLOduration=2.080989993 podStartE2EDuration="2.080989993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:50.080373764 +0000 UTC m=+1310.715525877" watchObservedRunningTime="2026-09-15 21:37:50.080989993 +0000 UTC m=+1310.716142106" Sep 15 21:37:50.733665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.733634 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:50.738730 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.738706 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:50.778721 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.778688 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:50.778966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.778943 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 15 21:37:50.779024 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:50.778990 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 15 21:37:51.044138 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:51.044058 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:51.045298 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:51.045274 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:37:51.779638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:51.779598 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 15 21:37:51.779814 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:51.779680 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 15 21:37:52.779442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:52.779409 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 15 21:37:52.779849 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:52.779460 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 15 21:37:53.779625 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:53.779590 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 15 21:37:53.780014 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:53.779642 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 15 21:37:54.779691 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:54.779657 2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 15 21:37:54.780112 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:54.779709 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 15 21:37:55.782236 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:55.782206 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:55.782688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:55.782673 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:37:55.783394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:37:55.783375 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:38:00.080620 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:00.080582 2552 scope.go:117] "RemoveContainer" containerID="658589c6d68e8a052f11083dd7153c7d666f94fdf0aa76ca66ec9a854f17702c" Sep 15 21:38:00.089597 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:00.089573 2552 scope.go:117] "RemoveContainer" containerID="18d2e839eeb5d4b0ec93017e3c76ade276958b9309da45f79385c6e025efcf56" Sep 15 21:38:00.097859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:00.097840 2552 scope.go:117] "RemoveContainer" containerID="4e2c94b1bec640ef2aff431d886574ca86e2ec7ff77973d61439ab28f9a67b92" Sep 15 21:38:27.297538 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.297474 2552 generic.go:356] "Generic (PLEG): container finished" podID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerID="a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c" exitCode=0 Sep 15 21:38:27.297907 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.297548 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerDied","Data":"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c"} Sep 15 21:38:27.298027 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.298013 2552 scope.go:117] "RemoveContainer" containerID="a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c" Sep 15 21:38:27.299283 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.299256 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:38:27.299345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.299331 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:38:27.299381 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:27.299364 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:38:28.305533 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:28.305495 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerStarted","Data":"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a"} Sep 15 21:38:28.733498 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:28.733468 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:38:28.733661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:28.733631 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:38:28.733709 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:28.733672 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 15 21:38:29.734308 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:29.734273 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:38:29.734713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:29.734329 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 15 21:38:30.734283 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:30.734248 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:38:30.734457 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:30.734303 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 15 21:38:31.734032 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:31.733994 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:38:31.734233 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:31.734051 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 15 21:38:32.733772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:32.733724 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 15 21:38:32.734187 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:32.733796 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 15 21:38:33.737575 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:33.737545 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:38:33.737989 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:33.737809 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:38:33.738580 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:38:33.738559 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:39:18.462380 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.462337 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79"] Sep 15 21:39:18.465119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.465105 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:18.467391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.467375 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:39:18.473227 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.473208 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79"] Sep 15 21:39:18.527015 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.526991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-56820a3c-h6c79\" (UID: \"c4ff3889-3baa-4308-ad74-f62611fd5c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:18.627988 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.627964 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-56820a3c-h6c79\" (UID: \"c4ff3889-3baa-4308-ad74-f62611fd5c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:18.628528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.628509 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-56820a3c-h6c79\" (UID: \"c4ff3889-3baa-4308-ad74-f62611fd5c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:18.777132 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:18.777065 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:19.060747 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:19.060718 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79"] Sep 15 21:39:19.062360 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:39:19.062330 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ff3889_3baa_4308_ad74_f62611fd5c03.slice/crio-8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598 WatchSource:0}: Error finding container 8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598: Status 404 returned error can't find the container with id 8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598 Sep 15 21:39:19.622780 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:19.622745 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" event={"ID":"c4ff3889-3baa-4308-ad74-f62611fd5c03","Type":"ContainerStarted","Data":"a1fdcc6ddb59a304d0018029c0589259a9a8d32cbce4c8349dcf2ab765cf2c10"} Sep 15 21:39:19.623158 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:19.622823 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" event={"ID":"c4ff3889-3baa-4308-ad74-f62611fd5c03","Type":"ContainerStarted","Data":"8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598"} Sep 15 21:39:19.644073 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:19.644006 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" podStartSLOduration=1.6439856229999998 podStartE2EDuration="1.643985623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:19.639116472 +0000 UTC m=+1400.274268584" watchObservedRunningTime="2026-09-15 21:39:19.643985623 +0000 UTC m=+1400.279137735" Sep 15 21:39:20.630927 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:20.630892 2552 generic.go:356] "Generic (PLEG): container finished" podID="c4ff3889-3baa-4308-ad74-f62611fd5c03" containerID="a1fdcc6ddb59a304d0018029c0589259a9a8d32cbce4c8349dcf2ab765cf2c10" exitCode=0 Sep 15 21:39:20.631301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:20.630934 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" event={"ID":"c4ff3889-3baa-4308-ad74-f62611fd5c03","Type":"ContainerDied","Data":"a1fdcc6ddb59a304d0018029c0589259a9a8d32cbce4c8349dcf2ab765cf2c10"} Sep 15 21:39:21.767795 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:21.767768 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:21.857234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:21.857208 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca\") pod \"c4ff3889-3baa-4308-ad74-f62611fd5c03\" (UID: \"c4ff3889-3baa-4308-ad74-f62611fd5c03\") " Sep 15 21:39:21.857506 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:21.857485 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4ff3889-3baa-4308-ad74-f62611fd5c03" (UID: "c4ff3889-3baa-4308-ad74-f62611fd5c03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:39:21.958889 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:21.958806 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ff3889-3baa-4308-ad74-f62611fd5c03-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:39:22.644402 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:22.644372 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" Sep 15 21:39:22.644593 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:22.644372 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79" event={"ID":"c4ff3889-3baa-4308-ad74-f62611fd5c03","Type":"ContainerDied","Data":"8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598"} Sep 15 21:39:22.644593 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:22.644572 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4e341eb4ff149eebef28429ecd91e9b12e1713d382339e79acf37afe63e598" Sep 15 21:39:34.161276 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.161240 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:39:34.161664 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.161483 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" podUID="77bbe11e-e931-400f-815d-6872cead0df2" containerName="payload-processing" containerID="cri-o://8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64" gracePeriod=30 Sep 15 21:39:34.355043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.355010 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79"] Sep 15 21:39:34.362044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.362015 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-56820a3c-h6c79"] Sep 15 21:39:34.418914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.418867 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:39:34.571767 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.571718 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume\") pod \"77bbe11e-e931-400f-815d-6872cead0df2\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " Sep 15 21:39:34.571989 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.571787 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwczt\" (UniqueName: \"kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt\") pod \"77bbe11e-e931-400f-815d-6872cead0df2\" (UID: \"77bbe11e-e931-400f-815d-6872cead0df2\") " Sep 15 21:39:34.572159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.572135 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77bbe11e-e931-400f-815d-6872cead0df2" (UID: "77bbe11e-e931-400f-815d-6872cead0df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:39:34.573742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.573721 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt" (OuterVolumeSpecName: "kube-api-access-qwczt") pod "77bbe11e-e931-400f-815d-6872cead0df2" (UID: "77bbe11e-e931-400f-815d-6872cead0df2"). InnerVolumeSpecName "kube-api-access-qwczt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:39:34.673366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.673302 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bbe11e-e931-400f-815d-6872cead0df2-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.673366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.673332 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwczt\" (UniqueName: \"kubernetes.io/projected/77bbe11e-e931-400f-815d-6872cead0df2-kube-api-access-qwczt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.718865 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.718824 2552 generic.go:356] "Generic (PLEG): container finished" podID="77bbe11e-e931-400f-815d-6872cead0df2" containerID="8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64" exitCode=0 Sep 15 21:39:34.718944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.718888 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" Sep 15 21:39:34.718944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.718909 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" event={"ID":"77bbe11e-e931-400f-815d-6872cead0df2","Type":"ContainerDied","Data":"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64"} Sep 15 21:39:34.719040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.718950 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm" event={"ID":"77bbe11e-e931-400f-815d-6872cead0df2","Type":"ContainerDied","Data":"d4572b5d2f5580f5db06f7a79d4c8bfe77e655c0709ddd324729e014f4a2269f"} Sep 15 21:39:34.719040 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.718971 2552 scope.go:117] "RemoveContainer" containerID="8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64" Sep 15 21:39:34.728282 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.728264 2552 scope.go:117] "RemoveContainer" containerID="8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64" Sep 15 21:39:34.728556 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:39:34.728536 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64\": container with ID starting with 8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64 not found: ID does not exist" containerID="8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64" Sep 15 21:39:34.728616 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.728568 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64"} err="failed to get container status \"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64\": rpc error: code = NotFound desc = could not find container \"8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64\": container with ID starting with 8316ce9f6ef7a5aa61ffb65dd81a8c5ac05f91337f5977c4c511868ab73a0f64 not found: ID does not exist" Sep 15 21:39:34.743525 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.743504 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:39:34.746106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:34.746085 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-56820a3c-689ff6754f-6ctcm"] Sep 15 21:39:35.797486 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:35.797454 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bbe11e-e931-400f-815d-6872cead0df2" path="/var/lib/kubelet/pods/77bbe11e-e931-400f-815d-6872cead0df2/volumes" Sep 15 21:39:35.797908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:39:35.797799 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ff3889-3baa-4308-ad74-f62611fd5c03" path="/var/lib/kubelet/pods/c4ff3889-3baa-4308-ad74-f62611fd5c03/volumes" Sep 15 21:40:59.904562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:40:59.904534 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:40:59.911199 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:40:59.911175 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:41:00.158430 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:00.158353 2552 scope.go:117] "RemoveContainer" containerID="0e5ce46352032aa7debe95e3534518cb023aa358d892ea83b3889dc7c6a7c51c" Sep 15 21:41:19.670857 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.670808 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:41:19.671301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671227 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ff3889-3baa-4308-ad74-f62611fd5c03" containerName="revoke-keys" Sep 15 21:41:19.671301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671239 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ff3889-3baa-4308-ad74-f62611fd5c03" containerName="revoke-keys" Sep 15 21:41:19.671301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671255 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bbe11e-e931-400f-815d-6872cead0df2" containerName="payload-processing" Sep 15 21:41:19.671301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671261 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bbe11e-e931-400f-815d-6872cead0df2" containerName="payload-processing" Sep 15 21:41:19.671436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671331 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bbe11e-e931-400f-815d-6872cead0df2" containerName="payload-processing" Sep 15 21:41:19.671436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.671338 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ff3889-3baa-4308-ad74-f62611fd5c03" containerName="revoke-keys" Sep 15 21:41:19.675660 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.675641 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.678411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.678388 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0aa2e-openshift-default-dockercfg-ql7sz\"" Sep 15 21:41:19.690681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.690660 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:41:19.797592 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.797551 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.797887 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.797864 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798048 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798029 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798209 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798183 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798354 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798340 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798498 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798483 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798630 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798618 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798726 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.798907 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.798890 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlml\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900027 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.899980 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900226 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900063 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900226 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900150 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900226 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900211 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900235 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900281 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlml\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900379 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900426 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.900559 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.900466 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.901206 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.901178 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.901372 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.901304 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.901549 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.901525 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.902291 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.901687 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.902427 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.902161 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.904541 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.904514 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.904541 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.904527 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.909730 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.909697 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.910129 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.910110 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlml\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml\") pod \"e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:19.986998 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:19.986941 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:20.265193 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.265165 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:41:20.266039 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:41:20.266012 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800831c4_b087_4663_ba27_e88703c465c2.slice/crio-9e6f389d8a638a13580ba4bb8bd598f974e9a176b3cf3c4549da23a5dddc75fd WatchSource:0}: Error finding container 9e6f389d8a638a13580ba4bb8bd598f974e9a176b3cf3c4549da23a5dddc75fd: Status 404 returned error can't find the container with id 9e6f389d8a638a13580ba4bb8bd598f974e9a176b3cf3c4549da23a5dddc75fd Sep 15 21:41:20.268122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.268090 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:20.268218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.268166 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:20.268218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.268211 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:20.396375 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.396335 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerStarted","Data":"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a"} Sep 15 21:41:20.396562 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.396402 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerStarted","Data":"9e6f389d8a638a13580ba4bb8bd598f974e9a176b3cf3c4549da23a5dddc75fd"} Sep 15 21:41:20.420455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.420399 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podStartSLOduration=1.420385427 podStartE2EDuration="1.420385427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:20.417140255 +0000 UTC m=+1521.052292369" watchObservedRunningTime="2026-09-15 21:41:20.420385427 +0000 UTC m=+1521.055537538" Sep 15 21:41:20.987436 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:20.987402 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:21.993263 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:21.993232 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:22.408402 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:22.408371 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:22.409370 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:22.409349 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:46.422036 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.422004 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:41:46.425705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.425676 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.428161 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.428136 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"embedding-w0-b99d7b-kserve-self-signed-certs\"" Sep 15 21:41:46.437189 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.437161 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:41:46.546705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546670 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.546901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqkb\" (UniqueName: \"kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.546901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546744 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.546901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546766 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.546901 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.547056 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.546919 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.647715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647686 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.647920 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647722 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqkb\" (UniqueName: \"kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.647920 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647753 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.647920 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647856 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.648107 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647955 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.648107 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.647997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.648336 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.648307 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.648336 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.648321 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.648494 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.648359 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.649910 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.649889 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.650335 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.650320 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.655675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.655659 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqkb\" (UniqueName: \"kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb\") pod \"embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:46.736043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:46.736019 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:47.011315 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.011288 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:41:47.012722 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:41:47.012697 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91155c4a_e0dd_4bfb_81da_a6456b26a13c.slice/crio-6fd33d24bb51479c5ed5bf4ad9a42372cc258d5360b0b9cc9f86b87140dbb1c5 WatchSource:0}: Error finding container 6fd33d24bb51479c5ed5bf4ad9a42372cc258d5360b0b9cc9f86b87140dbb1c5: Status 404 returned error can't find the container with id 6fd33d24bb51479c5ed5bf4ad9a42372cc258d5360b0b9cc9f86b87140dbb1c5 Sep 15 21:41:47.582961 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.582928 2552 generic.go:356] "Generic (PLEG): container finished" podID="800831c4-b087-4663-ba27-e88703c465c2" containerID="403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a" exitCode=0 Sep 15 21:41:47.583403 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.582996 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerDied","Data":"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a"} Sep 15 21:41:47.583454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.583410 2552 scope.go:117] "RemoveContainer" containerID="403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a" Sep 15 21:41:47.584653 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.584619 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:47.584773 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.584703 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:47.584866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.584779 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:41:47.584866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.584787 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" event={"ID":"91155c4a-e0dd-4bfb-81da-a6456b26a13c","Type":"ContainerStarted","Data":"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d"} Sep 15 21:41:47.584866 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.584822 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" event={"ID":"91155c4a-e0dd-4bfb-81da-a6456b26a13c","Type":"ContainerStarted","Data":"6fd33d24bb51479c5ed5bf4ad9a42372cc258d5360b0b9cc9f86b87140dbb1c5"} Sep 15 21:41:47.624859 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:47.622535 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" podStartSLOduration=1.622506328 podStartE2EDuration="1.622506328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:47.617497044 +0000 UTC m=+1548.252649152" watchObservedRunningTime="2026-09-15 21:41:47.622506328 +0000 UTC m=+1548.257658439" Sep 15 21:41:48.592792 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:48.592757 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerStarted","Data":"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7"} Sep 15 21:41:48.987914 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:48.987875 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:48.988813 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:48.988792 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 15 21:41:48.988905 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:48.988865 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 15 21:41:49.988155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:49.988119 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 15 21:41:49.988556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:49.988186 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 15 21:41:50.987647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:50.987612 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 15 21:41:50.987813 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:50.987665 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 15 21:41:51.988316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:51.988276 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 15 21:41:51.989449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:51.989414 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 15 21:41:52.408819 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:52.408788 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:52.988010 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:52.987964 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 15 21:41:52.988179 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:52.988043 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 15 21:41:53.992686 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:53.992655 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:53.994601 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:53.994575 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:41:56.736681 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:56.736651 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:56.737122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:56.736705 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:56.749243 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:56.749217 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:41:57.666096 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:41:57.666067 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:42:00.184063 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:00.184032 2552 scope.go:117] "RemoveContainer" containerID="b3bf049df8405f779d3d22608338bcf4f7ebb2053b4861680cbea96a1c0bb7ec" Sep 15 21:42:03.275511 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.275478 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:42:03.277940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.277924 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.280495 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.280470 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-57a27b\"" Sep 15 21:42:03.280620 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.280575 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-57a27b-dockercfg-g6pdp\"" Sep 15 21:42:03.290687 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.290664 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:42:03.314235 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.314212 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:42:03.316601 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.316587 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.325815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.325796 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:42:03.412469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.412434 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.412647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.412628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4zl\" (UniqueName: \"kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.412755 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.412732 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.412894 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.412870 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9p4\" (UniqueName: \"kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.513626 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.513592 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.513793 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.513636 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4zl\" (UniqueName: \"kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.513793 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.513673 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.513793 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.513720 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9p4\" (UniqueName: \"kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.514418 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.514394 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.514500 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.514428 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.521806 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.521787 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4zl\" (UniqueName: \"kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl\") pod \"payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.522326 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.522308 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9p4\" (UniqueName: \"kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4\") pod \"payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.588598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.588538 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:03.628028 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.627736 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:03.897485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.897457 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:42:03.898952 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:42:03.898927 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fac440f_0963_4659_8eda_2ad2d801814f.slice/crio-3f1ec828be77eea37107ce2c1d708b0017d2ac4dcf5b04230992a42f5a7676d7 WatchSource:0}: Error finding container 3f1ec828be77eea37107ce2c1d708b0017d2ac4dcf5b04230992a42f5a7676d7: Status 404 returned error can't find the container with id 3f1ec828be77eea37107ce2c1d708b0017d2ac4dcf5b04230992a42f5a7676d7 Sep 15 21:42:03.957030 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:03.956469 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:42:03.964326 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:42:03.964285 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2696009_763d_4899_b7dd_54b455f53e10.slice/crio-393bcde508762c87561d628f736273882ee3bdeee0491803202d8a60068e0b7b WatchSource:0}: Error finding container 393bcde508762c87561d628f736273882ee3bdeee0491803202d8a60068e0b7b: Status 404 returned error can't find the container with id 393bcde508762c87561d628f736273882ee3bdeee0491803202d8a60068e0b7b Sep 15 21:42:04.706331 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.706248 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" event={"ID":"a2696009-763d-4899-b7dd-54b455f53e10","Type":"ContainerStarted","Data":"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c"} Sep 15 21:42:04.706331 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.706323 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" event={"ID":"a2696009-763d-4899-b7dd-54b455f53e10","Type":"ContainerStarted","Data":"393bcde508762c87561d628f736273882ee3bdeee0491803202d8a60068e0b7b"} Sep 15 21:42:04.706873 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.706356 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:04.708218 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.708195 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" event={"ID":"8fac440f-0963-4659-8eda-2ad2d801814f","Type":"ContainerStarted","Data":"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2"} Sep 15 21:42:04.708345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.708234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" event={"ID":"8fac440f-0963-4659-8eda-2ad2d801814f","Type":"ContainerStarted","Data":"3f1ec828be77eea37107ce2c1d708b0017d2ac4dcf5b04230992a42f5a7676d7"} Sep 15 21:42:04.708345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.708261 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:04.725597 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.725557 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" podStartSLOduration=1.725543052 podStartE2EDuration="1.725543052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:04.723143268 +0000 UTC m=+1565.358295369" watchObservedRunningTime="2026-09-15 21:42:04.725543052 +0000 UTC m=+1565.360695220" Sep 15 21:42:04.743602 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:04.743564 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" podStartSLOduration=1.743549856 podStartE2EDuration="1.743549856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:04.739253272 +0000 UTC m=+1565.374405384" watchObservedRunningTime="2026-09-15 21:42:04.743549856 +0000 UTC m=+1565.378701966" Sep 15 21:42:15.717984 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:15.717950 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:42:15.718554 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:15.718032 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:42:29.222791 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.222755 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp"] Sep 15 21:42:29.226092 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.226072 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:29.228435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.228413 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:42:29.233918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.233893 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp"] Sep 15 21:42:29.247741 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.247711 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp\" (UID: \"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:29.348965 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.348938 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp\" (UID: \"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:29.349541 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.349522 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp\" (UID: \"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:29.539409 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.539328 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:29.811321 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.811297 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp"] Sep 15 21:42:29.813184 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:42:29.813156 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae73e8_b2b4_43c3_a0c6_994f4fbbf918.slice/crio-8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90 WatchSource:0}: Error finding container 8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90: Status 404 returned error can't find the container with id 8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90 Sep 15 21:42:29.869146 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:29.869117 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" event={"ID":"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918","Type":"ContainerStarted","Data":"8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90"} Sep 15 21:42:30.876492 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:30.876455 2552 generic.go:356] "Generic (PLEG): container finished" podID="79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" containerID="204895fe0e6be0f717dc9f282a09ef8095360cb5cb0cbdc8eb47d40d02b69fba" exitCode=0 Sep 15 21:42:30.876845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:30.876540 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" event={"ID":"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918","Type":"ContainerDied","Data":"204895fe0e6be0f717dc9f282a09ef8095360cb5cb0cbdc8eb47d40d02b69fba"} Sep 15 21:42:32.023992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.023968 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:32.075116 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.075092 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca\") pod \"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918\" (UID: \"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918\") " Sep 15 21:42:32.075420 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.075399 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" (UID: "79ae73e8-b2b4-43c3-a0c6-994f4fbbf918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:32.176635 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.176572 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:32.889940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.889910 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" Sep 15 21:42:32.890105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.889946 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp" event={"ID":"79ae73e8-b2b4-43c3-a0c6-994f4fbbf918","Type":"ContainerDied","Data":"8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90"} Sep 15 21:42:32.890105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:32.889995 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c28c4bfc9d2fafeb60c81b422a14404a7ca037fa4461da7a240787c7d2a8f90" Sep 15 21:42:46.250505 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:46.250472 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp"] Sep 15 21:42:46.253257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:46.253230 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0aa2e-z97hp"] Sep 15 21:42:47.797447 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:47.797416 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" path="/var/lib/kubelet/pods/79ae73e8-b2b4-43c3-a0c6-994f4fbbf918/volumes" Sep 15 21:42:53.097870 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.097788 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:42:53.098224 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.098010 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" containerID="cri-o://f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7" gracePeriod=30 Sep 15 21:42:53.384374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.384344 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2"] Sep 15 21:42:53.384777 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.384763 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" containerName="revoke-keys" Sep 15 21:42:53.384824 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.384778 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" containerName="revoke-keys" Sep 15 21:42:53.384906 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.384894 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ae73e8-b2b4-43c3-a0c6-994f4fbbf918" containerName="revoke-keys" Sep 15 21:42:53.395069 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.395038 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:53.398042 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.398014 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:42:53.398177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.398102 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2"] Sep 15 21:42:53.483897 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.483865 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2\" (UID: \"b67e6826-7bcd-40a3-ade1-1c80409587c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:53.584771 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.584747 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2\" (UID: \"b67e6826-7bcd-40a3-ade1-1c80409587c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:53.585433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.585415 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2\" (UID: \"b67e6826-7bcd-40a3-ade1-1c80409587c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:53.707284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.707202 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:53.983380 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.983352 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2"] Sep 15 21:42:53.984325 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:42:53.984294 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67e6826_7bcd_40a3_ade1_1c80409587c4.slice/crio-17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef WatchSource:0}: Error finding container 17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef: Status 404 returned error can't find the container with id 17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef Sep 15 21:42:53.995929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.995903 2552 patch_prober.go:28] interesting pod/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:42:53.996026 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:53.995951 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:42:54.020893 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:54.020858 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" event={"ID":"b67e6826-7bcd-40a3-ade1-1c80409587c4","Type":"ContainerStarted","Data":"17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef"} Sep 15 21:42:55.029103 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:55.029072 2552 generic.go:356] "Generic (PLEG): container finished" podID="b67e6826-7bcd-40a3-ade1-1c80409587c4" containerID="42c16a60eddc12b898f2caf9e01341fcde22c0b0bea53d73f4e9e838e3b56b4c" exitCode=0 Sep 15 21:42:55.029507 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:55.029159 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" event={"ID":"b67e6826-7bcd-40a3-ade1-1c80409587c4","Type":"ContainerDied","Data":"42c16a60eddc12b898f2caf9e01341fcde22c0b0bea53d73f4e9e838e3b56b4c"} Sep 15 21:42:56.175444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:56.175418 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:56.309645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:56.309560 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca\") pod \"b67e6826-7bcd-40a3-ade1-1c80409587c4\" (UID: \"b67e6826-7bcd-40a3-ade1-1c80409587c4\") " Sep 15 21:42:56.309956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:56.309934 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b67e6826-7bcd-40a3-ade1-1c80409587c4" (UID: "b67e6826-7bcd-40a3-ade1-1c80409587c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:56.411126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:56.411100 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67e6826-7bcd-40a3-ade1-1c80409587c4-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:57.042925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:57.042892 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" event={"ID":"b67e6826-7bcd-40a3-ade1-1c80409587c4","Type":"ContainerDied","Data":"17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef"} Sep 15 21:42:57.042925 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:57.042912 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2" Sep 15 21:42:57.043130 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:57.042937 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a9877b76ce1df80b2e6915e199a2335e77133aac6e8eb67a356679f231bdef" Sep 15 21:42:58.361552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.361487 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:42:58.531962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.531931 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.531975 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532028 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532075 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532114 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zlml\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532150 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532230 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:58.532351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532233 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532298 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532351 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532346 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket\") pod \"800831c4-b087-4663-ba27-e88703c465c2\" (UID: \"800831c4-b087-4663-ba27-e88703c465c2\") " Sep 15 21:42:58.532610 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532449 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:58.532610 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532466 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:58.532724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532667 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:58.532782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532716 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:58.532782 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532771 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.532924 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532789 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.532924 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532803 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.532924 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.532816 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.534259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.534227 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:58.534367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.534327 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml" (OuterVolumeSpecName: "kube-api-access-2zlml") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "kube-api-access-2zlml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:58.534587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.534566 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:42:58.534587 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.534571 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "800831c4-b087-4663-ba27-e88703c465c2" (UID: "800831c4-b087-4663-ba27-e88703c465c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:58.633940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.633911 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.633940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.633937 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlml\" (UniqueName: \"kubernetes.io/projected/800831c4-b087-4663-ba27-e88703c465c2-kube-api-access-2zlml\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.634105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.633952 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800831c4-b087-4663-ba27-e88703c465c2-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.634105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.633964 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800831c4-b087-4663-ba27-e88703c465c2-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:58.634105 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:58.633975 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800831c4-b087-4663-ba27-e88703c465c2-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:42:59.058449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.058365 2552 generic.go:356] "Generic (PLEG): container finished" podID="800831c4-b087-4663-ba27-e88703c465c2" containerID="f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7" exitCode=0 Sep 15 21:42:59.058449 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.058409 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" Sep 15 21:42:59.058680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.058443 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerDied","Data":"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7"} Sep 15 21:42:59.058680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.058501 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7" event={"ID":"800831c4-b087-4663-ba27-e88703c465c2","Type":"ContainerDied","Data":"9e6f389d8a638a13580ba4bb8bd598f974e9a176b3cf3c4549da23a5dddc75fd"} Sep 15 21:42:59.058680 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.058532 2552 scope.go:117] "RemoveContainer" containerID="f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7" Sep 15 21:42:59.068727 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.068712 2552 scope.go:117] "RemoveContainer" containerID="403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a" Sep 15 21:42:59.077274 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.077251 2552 scope.go:117] "RemoveContainer" containerID="f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7" Sep 15 21:42:59.077596 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:42:59.077573 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7\": container with ID starting with f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7 not found: ID does not exist" containerID="f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7" Sep 15 21:42:59.077676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.077610 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7"} err="failed to get container status \"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7\": rpc error: code = NotFound desc = could not find container \"f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7\": container with ID starting with f8410392e297f216c5cf31de2278594711715c96e7645c36fef4910b9202d6c7 not found: ID does not exist" Sep 15 21:42:59.077676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.077628 2552 scope.go:117] "RemoveContainer" containerID="403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a" Sep 15 21:42:59.078029 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:42:59.078001 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a\": container with ID starting with 403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a not found: ID does not exist" containerID="403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a" Sep 15 21:42:59.078135 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.078036 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a"} err="failed to get container status \"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a\": rpc error: code = NotFound desc = could not find container \"403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a\": container with ID starting with 403dfd54560160f598160b69d98fef3b7525798b5381a905cdf55ccc235a792a not found: ID does not exist" Sep 15 21:42:59.081107 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.081085 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:42:59.088008 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.087984 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0aa2e-openshift-default-59f7f787fd-4hqs7"] Sep 15 21:42:59.791701 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:42:59.791670 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="800831c4-b087-4663-ba27-e88703c465c2" path="/var/lib/kubelet/pods/800831c4-b087-4663-ba27-e88703c465c2/volumes" Sep 15 21:43:00.208038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:00.208008 2552 scope.go:117] "RemoveContainer" containerID="348bd5855aed0c9376bffaeb44f33a54a56b972257cbf384fd7bed9a6219ca29" Sep 15 21:43:00.218300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:00.218277 2552 scope.go:117] "RemoveContainer" containerID="97656c9df9a6b7701dd481f8b627f500a0315e147e27ba35160c4a981cad7a0b" Sep 15 21:43:09.043710 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.043673 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:43:09.044988 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.044929 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" podUID="8fac440f-0963-4659-8eda-2ad2d801814f" containerName="payload-processing" containerID="cri-o://9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2" gracePeriod=30 Sep 15 21:43:09.050384 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.050357 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:43:09.050608 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.050563 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" podUID="a2696009-763d-4899-b7dd-54b455f53e10" containerName="payload-pre-processing" containerID="cri-o://a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c" gracePeriod=30 Sep 15 21:43:09.217865 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.217241 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2"] Sep 15 21:43:09.220922 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.220900 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57a27b-jj9j2"] Sep 15 21:43:09.328239 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.328218 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:43:09.338952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.338935 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:43:09.427302 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427265 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume\") pod \"a2696009-763d-4899-b7dd-54b455f53e10\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " Sep 15 21:43:09.427499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427333 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg4zl\" (UniqueName: \"kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl\") pod \"a2696009-763d-4899-b7dd-54b455f53e10\" (UID: \"a2696009-763d-4899-b7dd-54b455f53e10\") " Sep 15 21:43:09.427499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427378 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk9p4\" (UniqueName: \"kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4\") pod \"8fac440f-0963-4659-8eda-2ad2d801814f\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " Sep 15 21:43:09.427599 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427546 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume\") pod \"8fac440f-0963-4659-8eda-2ad2d801814f\" (UID: \"8fac440f-0963-4659-8eda-2ad2d801814f\") " Sep 15 21:43:09.427655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427611 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2696009-763d-4899-b7dd-54b455f53e10" (UID: "a2696009-763d-4899-b7dd-54b455f53e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:09.427929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427905 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fac440f-0963-4659-8eda-2ad2d801814f" (UID: "8fac440f-0963-4659-8eda-2ad2d801814f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:09.428035 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.427948 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2696009-763d-4899-b7dd-54b455f53e10-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.429424 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.429399 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4" (OuterVolumeSpecName: "kube-api-access-jk9p4") pod "8fac440f-0963-4659-8eda-2ad2d801814f" (UID: "8fac440f-0963-4659-8eda-2ad2d801814f"). InnerVolumeSpecName "kube-api-access-jk9p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:09.429558 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.429497 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl" (OuterVolumeSpecName: "kube-api-access-jg4zl") pod "a2696009-763d-4899-b7dd-54b455f53e10" (UID: "a2696009-763d-4899-b7dd-54b455f53e10"). InnerVolumeSpecName "kube-api-access-jg4zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:09.529368 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.529343 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4zl\" (UniqueName: \"kubernetes.io/projected/a2696009-763d-4899-b7dd-54b455f53e10-kube-api-access-jg4zl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.529368 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.529365 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9p4\" (UniqueName: \"kubernetes.io/projected/8fac440f-0963-4659-8eda-2ad2d801814f-kube-api-access-jk9p4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.529513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.529374 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac440f-0963-4659-8eda-2ad2d801814f-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.791641 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:09.791609 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b67e6826-7bcd-40a3-ade1-1c80409587c4" path="/var/lib/kubelet/pods/b67e6826-7bcd-40a3-ade1-1c80409587c4/volumes" Sep 15 21:43:10.129142 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.129103 2552 generic.go:356] "Generic (PLEG): container finished" podID="8fac440f-0963-4659-8eda-2ad2d801814f" containerID="9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2" exitCode=0 Sep 15 21:43:10.129547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.129147 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" Sep 15 21:43:10.129547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.129188 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" event={"ID":"8fac440f-0963-4659-8eda-2ad2d801814f","Type":"ContainerDied","Data":"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2"} Sep 15 21:43:10.129547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.129247 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s" event={"ID":"8fac440f-0963-4659-8eda-2ad2d801814f","Type":"ContainerDied","Data":"3f1ec828be77eea37107ce2c1d708b0017d2ac4dcf5b04230992a42f5a7676d7"} Sep 15 21:43:10.129547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.129273 2552 scope.go:117] "RemoveContainer" containerID="9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2" Sep 15 21:43:10.130896 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.130869 2552 generic.go:356] "Generic (PLEG): container finished" podID="a2696009-763d-4899-b7dd-54b455f53e10" containerID="a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c" exitCode=0 Sep 15 21:43:10.131007 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.130907 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" Sep 15 21:43:10.131007 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.130930 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" event={"ID":"a2696009-763d-4899-b7dd-54b455f53e10","Type":"ContainerDied","Data":"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c"} Sep 15 21:43:10.131007 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.130976 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6" event={"ID":"a2696009-763d-4899-b7dd-54b455f53e10","Type":"ContainerDied","Data":"393bcde508762c87561d628f736273882ee3bdeee0491803202d8a60068e0b7b"} Sep 15 21:43:10.139216 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.139200 2552 scope.go:117] "RemoveContainer" containerID="9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2" Sep 15 21:43:10.139469 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:10.139452 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2\": container with ID starting with 9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2 not found: ID does not exist" containerID="9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2" Sep 15 21:43:10.139517 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.139475 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2"} err="failed to get container status \"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2\": rpc error: code = NotFound desc = could not find container \"9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2\": container with ID starting with 9474250c0d65894a4dd11a6257fd866770b9b3ffc2a264590837db21116ffbd2 not found: ID does not exist" Sep 15 21:43:10.139517 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.139489 2552 scope.go:117] "RemoveContainer" containerID="a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c" Sep 15 21:43:10.147756 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.147737 2552 scope.go:117] "RemoveContainer" containerID="a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c" Sep 15 21:43:10.148024 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:10.148006 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c\": container with ID starting with a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c not found: ID does not exist" containerID="a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c" Sep 15 21:43:10.148093 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.148030 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c"} err="failed to get container status \"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c\": rpc error: code = NotFound desc = could not find container \"a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c\": container with ID starting with a9db659db30204bef552fc134b0dd25d04f3e2ad6e2e8d3e80284f74edb2740c not found: ID does not exist" Sep 15 21:43:10.148431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.148413 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:43:10.152780 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.152752 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57a27b-66c94746fc-pv72s"] Sep 15 21:43:10.163742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.163721 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:43:10.167744 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:10.167723 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57a27b-5db4c7f6b-rkvz6"] Sep 15 21:43:11.790335 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:11.790299 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fac440f-0963-4659-8eda-2ad2d801814f" path="/var/lib/kubelet/pods/8fac440f-0963-4659-8eda-2ad2d801814f/volumes" Sep 15 21:43:11.790862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:11.790821 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2696009-763d-4899-b7dd-54b455f53e10" path="/var/lib/kubelet/pods/a2696009-763d-4899-b7dd-54b455f53e10/volumes" Sep 15 21:43:16.409728 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.409687 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx"] Sep 15 21:43:16.410127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410102 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b67e6826-7bcd-40a3-ade1-1c80409587c4" containerName="revoke-keys" Sep 15 21:43:16.410127 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410114 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b67e6826-7bcd-40a3-ade1-1c80409587c4" containerName="revoke-keys" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410133 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410139 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410150 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410155 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410162 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2696009-763d-4899-b7dd-54b455f53e10" containerName="payload-pre-processing" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410167 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2696009-763d-4899-b7dd-54b455f53e10" containerName="payload-pre-processing" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410178 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fac440f-0963-4659-8eda-2ad2d801814f" containerName="payload-processing" Sep 15 21:43:16.410202 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410183 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fac440f-0963-4659-8eda-2ad2d801814f" containerName="payload-processing" Sep 15 21:43:16.410445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410243 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410250 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b67e6826-7bcd-40a3-ade1-1c80409587c4" containerName="revoke-keys" Sep 15 21:43:16.410445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410260 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="800831c4-b087-4663-ba27-e88703c465c2" containerName="istio-proxy" Sep 15 21:43:16.410445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410266 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fac440f-0963-4659-8eda-2ad2d801814f" containerName="payload-processing" Sep 15 21:43:16.410445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.410273 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2696009-763d-4899-b7dd-54b455f53e10" containerName="payload-pre-processing" Sep 15 21:43:16.414893 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.414867 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:16.417375 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.417356 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:43:16.422118 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.422093 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx"] Sep 15 21:43:16.494120 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.494099 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx\" (UID: \"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:16.594928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.594902 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx\" (UID: \"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:16.595502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.595485 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx\" (UID: \"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:16.727038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:16.726974 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:17.001885 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:17.001860 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx"] Sep 15 21:43:17.003576 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:43:17.003553 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c638b5_a6f5_4d81_a4c8_9b347fff3dcb.slice/crio-11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b WatchSource:0}: Error finding container 11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b: Status 404 returned error can't find the container with id 11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b Sep 15 21:43:17.181265 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:17.181234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" event={"ID":"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb","Type":"ContainerStarted","Data":"11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b"} Sep 15 21:43:18.188756 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:18.188724 2552 generic.go:356] "Generic (PLEG): container finished" podID="01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" containerID="a1ba5f6fb3096ee1766d74a055a6d5c0a82c00cc9d46c2488a1e6a9552202f48" exitCode=0 Sep 15 21:43:18.189132 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:18.188794 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" event={"ID":"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb","Type":"ContainerDied","Data":"a1ba5f6fb3096ee1766d74a055a6d5c0a82c00cc9d46c2488a1e6a9552202f48"} Sep 15 21:43:19.334177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:19.334152 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:19.423912 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:19.423879 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca\") pod \"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb\" (UID: \"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb\") " Sep 15 21:43:19.424195 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:19.424175 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" (UID: "01c638b5-a6f5-4d81-a4c8-9b347fff3dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:19.525198 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:19.525133 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:20.202039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:20.202008 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" event={"ID":"01c638b5-a6f5-4d81-a4c8-9b347fff3dcb","Type":"ContainerDied","Data":"11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b"} Sep 15 21:43:20.202216 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:20.202059 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bf7fb9b2a456515cfdc63657ecfcee8241f629308d1768467d1484e751c21b" Sep 15 21:43:20.202216 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:20.202014 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx" Sep 15 21:43:27.009210 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.009173 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:43:27.009646 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.009442 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" podUID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" containerName="payload-pre-processing" containerID="cri-o://ff9ae42bc4c5b2d565c0f7dacf7226c0e1dd6d35d377eab4fdddaea2afa16949" gracePeriod=30 Sep 15 21:43:27.212707 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.212225 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx"] Sep 15 21:43:27.221032 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.220976 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2d4fab-pkwvx"] Sep 15 21:43:27.247508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.247474 2552 generic.go:356] "Generic (PLEG): container finished" podID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" containerID="ff9ae42bc4c5b2d565c0f7dacf7226c0e1dd6d35d377eab4fdddaea2afa16949" exitCode=0 Sep 15 21:43:27.247676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.247547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" event={"ID":"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc","Type":"ContainerDied","Data":"ff9ae42bc4c5b2d565c0f7dacf7226c0e1dd6d35d377eab4fdddaea2afa16949"} Sep 15 21:43:27.290937 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.290910 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:43:27.398624 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.398593 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvnj8\" (UniqueName: \"kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8\") pod \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " Sep 15 21:43:27.398815 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.398647 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume\") pod \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\" (UID: \"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc\") " Sep 15 21:43:27.399038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.399015 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" (UID: "e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:27.400729 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.400708 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8" (OuterVolumeSpecName: "kube-api-access-jvnj8") pod "e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" (UID: "e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc"). InnerVolumeSpecName "kube-api-access-jvnj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:27.499876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.499852 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.499876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.499871 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnj8\" (UniqueName: \"kubernetes.io/projected/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc-kube-api-access-jvnj8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.790949 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:27.790917 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" path="/var/lib/kubelet/pods/01c638b5-a6f5-4d81-a4c8-9b347fff3dcb/volumes" Sep 15 21:43:28.256705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:28.256682 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" Sep 15 21:43:28.257143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:28.256724 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5" event={"ID":"e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc","Type":"ContainerDied","Data":"abebc1a75855d4428cd67055fbb640f3578d3d571a67323ba70a6d7018a581a8"} Sep 15 21:43:28.257143 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:28.256791 2552 scope.go:117] "RemoveContainer" containerID="ff9ae42bc4c5b2d565c0f7dacf7226c0e1dd6d35d377eab4fdddaea2afa16949" Sep 15 21:43:28.277259 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:28.277236 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:43:28.281240 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:28.281222 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2d4fab-77c6d8998-g4dl5"] Sep 15 21:43:29.791473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:29.791440 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" path="/var/lib/kubelet/pods/e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc/volumes" Sep 15 21:43:33.724440 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:33.724409 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:43:33.724949 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:33.724612 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" containerID="cri-o://a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330" gracePeriod=30 Sep 15 21:43:34.056197 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056113 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l"] Sep 15 21:43:34.056539 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056526 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" containerName="revoke-keys" Sep 15 21:43:34.056590 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056541 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" containerName="revoke-keys" Sep 15 21:43:34.056590 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056551 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" containerName="payload-pre-processing" Sep 15 21:43:34.056590 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056556 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" containerName="payload-pre-processing" Sep 15 21:43:34.056694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056613 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c638b5-a6f5-4d81-a4c8-9b347fff3dcb" containerName="revoke-keys" Sep 15 21:43:34.056694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.056623 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ef56b3-fddf-41b9-b0ea-5d86c0b202bc" containerName="payload-pre-processing" Sep 15 21:43:34.067159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.067127 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:34.067577 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.067533 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l"] Sep 15 21:43:34.069169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.069145 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:43:34.165997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.165970 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l\" (UID: \"f10ea75f-743b-43d0-bd22-7b470a796fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:34.267374 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.267346 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l\" (UID: \"f10ea75f-743b-43d0-bd22-7b470a796fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:34.267963 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.267945 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l\" (UID: \"f10ea75f-743b-43d0-bd22-7b470a796fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:34.379921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.379898 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:34.660706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:34.660674 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l"] Sep 15 21:43:34.661314 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:43:34.661288 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10ea75f_743b_43d0_bd22_7b470a796fd0.slice/crio-6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177 WatchSource:0}: Error finding container 6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177: Status 404 returned error can't find the container with id 6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177 Sep 15 21:43:35.302891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:35.302783 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" event={"ID":"f10ea75f-743b-43d0-bd22-7b470a796fd0","Type":"ContainerStarted","Data":"ad83929d784ebe3805704f8be73b499d2001c30325bf91c9303d66dbb2c5ae14"} Sep 15 21:43:35.302891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:35.302853 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" event={"ID":"f10ea75f-743b-43d0-bd22-7b470a796fd0","Type":"ContainerStarted","Data":"6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177"} Sep 15 21:43:35.320441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:35.320382 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" podStartSLOduration=1.320365286 podStartE2EDuration="1.320365286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:35.319382616 +0000 UTC m=+1655.954534727" watchObservedRunningTime="2026-09-15 21:43:35.320365286 +0000 UTC m=+1655.955517398" Sep 15 21:43:36.311028 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.310999 2552 generic.go:356] "Generic (PLEG): container finished" podID="f10ea75f-743b-43d0-bd22-7b470a796fd0" containerID="ad83929d784ebe3805704f8be73b499d2001c30325bf91c9303d66dbb2c5ae14" exitCode=0 Sep 15 21:43:36.311411 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.311084 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" event={"ID":"f10ea75f-743b-43d0-bd22-7b470a796fd0","Type":"ContainerDied","Data":"ad83929d784ebe3805704f8be73b499d2001c30325bf91c9303d66dbb2c5ae14"} Sep 15 21:43:36.638023 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.637996 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw"] Sep 15 21:43:36.641392 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.641375 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:36.650255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.650231 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw"] Sep 15 21:43:36.794434 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.794397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw\" (UID: \"6ac5ac98-79fb-474b-b43e-28983935339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:36.895677 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.895596 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw\" (UID: \"6ac5ac98-79fb-474b-b43e-28983935339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:36.896265 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.896245 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw\" (UID: \"6ac5ac98-79fb-474b-b43e-28983935339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:36.953552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:36.953528 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:37.225652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.225624 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw"] Sep 15 21:43:37.227181 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:43:37.227153 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac5ac98_79fb_474b_b43e_28983935339f.slice/crio-53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69 WatchSource:0}: Error finding container 53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69: Status 404 returned error can't find the container with id 53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69 Sep 15 21:43:37.318594 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.318559 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" event={"ID":"6ac5ac98-79fb-474b-b43e-28983935339f","Type":"ContainerStarted","Data":"53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69"} Sep 15 21:43:37.548774 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.548751 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:37.703653 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.703624 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca\") pod \"f10ea75f-743b-43d0-bd22-7b470a796fd0\" (UID: \"f10ea75f-743b-43d0-bd22-7b470a796fd0\") " Sep 15 21:43:37.704038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.704013 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f10ea75f-743b-43d0-bd22-7b470a796fd0" (UID: "f10ea75f-743b-43d0-bd22-7b470a796fd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:37.805392 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:37.805368 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10ea75f-743b-43d0-bd22-7b470a796fd0-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:38.325947 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.325912 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" event={"ID":"f10ea75f-743b-43d0-bd22-7b470a796fd0","Type":"ContainerDied","Data":"6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177"} Sep 15 21:43:38.325947 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.325938 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l" Sep 15 21:43:38.326455 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.325967 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bfd136c389769ece4ba2b01ced377c06762bac02f8b654c636b5b37c9595177" Sep 15 21:43:38.327530 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.327501 2552 generic.go:356] "Generic (PLEG): container finished" podID="6ac5ac98-79fb-474b-b43e-28983935339f" containerID="0896df76ed0278ac50abff8890f95d64a56b958df05995b84d5f7d5559b707bf" exitCode=0 Sep 15 21:43:38.327652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.327532 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" event={"ID":"6ac5ac98-79fb-474b-b43e-28983935339f","Type":"ContainerDied","Data":"0896df76ed0278ac50abff8890f95d64a56b958df05995b84d5f7d5559b707bf"} Sep 15 21:43:38.983755 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:38.983734 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:43:39.118898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.118862 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.118898 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.118904 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119131 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.118931 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119131 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119070 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq5pd\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119150 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119188 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119230 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119277 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119285 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:39.119400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119319 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert\") pod \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\" (UID: \"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b\") " Sep 15 21:43:39.119400 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119340 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:39.119606 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119521 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:39.119606 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119588 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:39.119715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119683 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.119715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119701 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.119850 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119715 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.119850 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119728 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.119958 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.119882 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:39.121458 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.121429 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:43:39.121607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.121575 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:39.121670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.121632 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd" (OuterVolumeSpecName: "kube-api-access-pq5pd") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "kube-api-access-pq5pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:39.121670 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.121645 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" (UID: "3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:39.221111 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.221088 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5pd\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-kube-api-access-pq5pd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.221111 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.221112 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.221264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.221122 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.221264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.221130 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.221264 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.221138 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.335435 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.335411 2552 generic.go:356] "Generic (PLEG): container finished" podID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerID="a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330" exitCode=0 Sep 15 21:43:39.335776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.335482 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" Sep 15 21:43:39.335776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.335488 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerDied","Data":"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330"} Sep 15 21:43:39.335776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.335561 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x" event={"ID":"3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b","Type":"ContainerDied","Data":"544e6341dc6102f5facadf1780e8ec43fe18c736fe3201ab0869410ad3484a80"} Sep 15 21:43:39.335776 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.335575 2552 scope.go:117] "RemoveContainer" containerID="a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330" Sep 15 21:43:39.346665 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.346640 2552 scope.go:117] "RemoveContainer" containerID="783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d" Sep 15 21:43:39.355532 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.355508 2552 scope.go:117] "RemoveContainer" containerID="a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330" Sep 15 21:43:39.355801 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:39.355782 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330\": container with ID starting with a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330 not found: ID does not exist" containerID="a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330" Sep 15 21:43:39.355874 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.355810 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330"} err="failed to get container status \"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330\": rpc error: code = NotFound desc = could not find container \"a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330\": container with ID starting with a37313e2c0899fd567d9a9ec2709cc6b6a56953a270c46b5016d97d52bb43330 not found: ID does not exist" Sep 15 21:43:39.355874 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.355851 2552 scope.go:117] "RemoveContainer" containerID="783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d" Sep 15 21:43:39.356134 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:39.356115 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d\": container with ID starting with 783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d not found: ID does not exist" containerID="783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d" Sep 15 21:43:39.356188 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.356139 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d"} err="failed to get container status \"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d\": rpc error: code = NotFound desc = could not find container \"783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d\": container with ID starting with 783b132ea2120231713b3aa8cbe102070f21146ca072273c4979c4a8410d169d not found: ID does not exist" Sep 15 21:43:39.363213 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.363191 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:43:39.368091 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.368069 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2d4fab-openshift-default-7dcfb54848-2ww6x"] Sep 15 21:43:39.487340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.487315 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:39.625241 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.625210 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca\") pod \"6ac5ac98-79fb-474b-b43e-28983935339f\" (UID: \"6ac5ac98-79fb-474b-b43e-28983935339f\") " Sep 15 21:43:39.625564 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.625542 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ac5ac98-79fb-474b-b43e-28983935339f" (UID: "6ac5ac98-79fb-474b-b43e-28983935339f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:39.726493 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.726429 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5ac98-79fb-474b-b43e-28983935339f-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:39.791281 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:39.791254 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" path="/var/lib/kubelet/pods/3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b/volumes" Sep 15 21:43:40.342960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:40.342932 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" Sep 15 21:43:40.343439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:40.342940 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw" event={"ID":"6ac5ac98-79fb-474b-b43e-28983935339f","Type":"ContainerDied","Data":"53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69"} Sep 15 21:43:40.343439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:40.343112 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53019492a88ed1a14377ea0987af8c0a87a769aa5be3b5bc356f822d9b4cfa69" Sep 15 21:43:41.523472 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:41.523434 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:43:41.532717 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:41.532692 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:43:45.197331 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:45.197296 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l"] Sep 15 21:43:45.199872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:45.199847 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb0f0e-8fx5l"] Sep 15 21:43:45.790764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:45.790729 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10ea75f-743b-43d0-bd22-7b470a796fd0" path="/var/lib/kubelet/pods/f10ea75f-743b-43d0-bd22-7b470a796fd0/volumes" Sep 15 21:43:51.797384 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:51.797353 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:43:51.797932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:51.797579 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" containerID="cri-o://7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7" gracePeriod=30 Sep 15 21:43:52.063805 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.063728 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz"] Sep 15 21:43:52.069160 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.069124 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b99d7b-29825130-jlfkz"] Sep 15 21:43:52.102986 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.102960 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:43:52.103225 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.103199 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" podUID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" containerName="payload-processing" containerID="cri-o://c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e" gracePeriod=30 Sep 15 21:43:52.326724 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.326650 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw"] Sep 15 21:43:52.334584 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.334547 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b99d7b-spbkw"] Sep 15 21:43:52.360932 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.360908 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:43:52.423502 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.423474 2552 generic.go:356] "Generic (PLEG): container finished" podID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" containerID="c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e" exitCode=0 Sep 15 21:43:52.423672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.423536 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" event={"ID":"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7","Type":"ContainerDied","Data":"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e"} Sep 15 21:43:52.423672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.423576 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" event={"ID":"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7","Type":"ContainerDied","Data":"ad8b2d2a4ba094add3e7bae4fce52f14934b8cb1a1fa73a6d5e54ebbd4184cd9"} Sep 15 21:43:52.423672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.423582 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns" Sep 15 21:43:52.423672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.423598 2552 scope.go:117] "RemoveContainer" containerID="c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e" Sep 15 21:43:52.432683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.432665 2552 scope.go:117] "RemoveContainer" containerID="c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e" Sep 15 21:43:52.432984 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:52.432961 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e\": container with ID starting with c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e not found: ID does not exist" containerID="c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e" Sep 15 21:43:52.433061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.432991 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e"} err="failed to get container status \"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e\": rpc error: code = NotFound desc = could not find container \"c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e\": container with ID starting with c4cce74e2803a87cfda392d7bc8bbe904adfafa98e9b89ccddf91de412f95f7e not found: ID does not exist" Sep 15 21:43:52.436412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.436383 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf5sg\" (UniqueName: \"kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg\") pod \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " Sep 15 21:43:52.441996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.436475 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume\") pod \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\" (UID: \"dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7\") " Sep 15 21:43:52.441996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.437231 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" (UID: "dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:52.441996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.441202 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg" (OuterVolumeSpecName: "kube-api-access-tf5sg") pod "dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" (UID: "dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7"). InnerVolumeSpecName "kube-api-access-tf5sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:52.537598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.537565 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5sg\" (UniqueName: \"kubernetes.io/projected/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-kube-api-access-tf5sg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:52.537598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.537591 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:52.746325 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.746298 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:43:52.754367 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.754341 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b99d7b-77c5fd8fd9-xvqns"] Sep 15 21:43:52.786504 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.786480 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:43:52.786707 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.786661 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" podUID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" containerName="payload-processing" containerID="cri-o://b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277" gracePeriod=30 Sep 15 21:43:52.805703 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.805667 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:43:52.806103 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:52.805949 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" podUID="e2da6711-1939-415c-9ae6-4536dbeed30d" containerName="payload-pre-processing" containerID="cri-o://4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22" gracePeriod=30 Sep 15 21:43:53.086013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.085899 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:43:53.102801 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.102777 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:43:53.143290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143257 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume\") pod \"e2da6711-1939-415c-9ae6-4536dbeed30d\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " Sep 15 21:43:53.143443 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143410 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bhfj\" (UniqueName: \"kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj\") pod \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " Sep 15 21:43:53.143510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143453 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7kkz\" (UniqueName: \"kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz\") pod \"e2da6711-1939-415c-9ae6-4536dbeed30d\" (UID: \"e2da6711-1939-415c-9ae6-4536dbeed30d\") " Sep 15 21:43:53.143510 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143484 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume\") pod \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\" (UID: \"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23\") " Sep 15 21:43:53.143645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143616 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2da6711-1939-415c-9ae6-4536dbeed30d" (UID: "e2da6711-1939-415c-9ae6-4536dbeed30d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:53.143847 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143807 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2da6711-1939-415c-9ae6-4536dbeed30d-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:53.143956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.143926 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" (UID: "8846e0f9-cc3e-4c43-86ec-7eb27ef4df23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:53.145607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.145580 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj" (OuterVolumeSpecName: "kube-api-access-8bhfj") pod "8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" (UID: "8846e0f9-cc3e-4c43-86ec-7eb27ef4df23"). InnerVolumeSpecName "kube-api-access-8bhfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:53.145689 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.145638 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz" (OuterVolumeSpecName: "kube-api-access-l7kkz") pod "e2da6711-1939-415c-9ae6-4536dbeed30d" (UID: "e2da6711-1939-415c-9ae6-4536dbeed30d"). InnerVolumeSpecName "kube-api-access-l7kkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:53.244733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.244667 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhfj\" (UniqueName: \"kubernetes.io/projected/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-kube-api-access-8bhfj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:53.244733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.244693 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kkz\" (UniqueName: \"kubernetes.io/projected/e2da6711-1939-415c-9ae6-4536dbeed30d-kube-api-access-l7kkz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:53.244733 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.244703 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:53.430948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.430921 2552 generic.go:356] "Generic (PLEG): container finished" podID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" containerID="b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277" exitCode=0 Sep 15 21:43:53.431113 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.430967 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" Sep 15 21:43:53.431113 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.431012 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" event={"ID":"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23","Type":"ContainerDied","Data":"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277"} Sep 15 21:43:53.431113 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.431072 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz" event={"ID":"8846e0f9-cc3e-4c43-86ec-7eb27ef4df23","Type":"ContainerDied","Data":"a8b0d1929901f456f71659c41d264a1d19cdc818597e85496b9fb5916680893a"} Sep 15 21:43:53.431113 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.431103 2552 scope.go:117] "RemoveContainer" containerID="b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277" Sep 15 21:43:53.433595 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.433571 2552 generic.go:356] "Generic (PLEG): container finished" podID="e2da6711-1939-415c-9ae6-4536dbeed30d" containerID="4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22" exitCode=0 Sep 15 21:43:53.433723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.433614 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" Sep 15 21:43:53.433723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.433629 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" event={"ID":"e2da6711-1939-415c-9ae6-4536dbeed30d","Type":"ContainerDied","Data":"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22"} Sep 15 21:43:53.433723 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.433668 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn" event={"ID":"e2da6711-1939-415c-9ae6-4536dbeed30d","Type":"ContainerDied","Data":"f7da071baa645eb6154b7f628a93905ef282b34c05debddd44b7afa7093c2d8e"} Sep 15 21:43:53.441722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.441706 2552 scope.go:117] "RemoveContainer" containerID="b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277" Sep 15 21:43:53.442119 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:53.441994 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277\": container with ID starting with b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277 not found: ID does not exist" containerID="b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277" Sep 15 21:43:53.442119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.442021 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277"} err="failed to get container status \"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277\": rpc error: code = NotFound desc = could not find container \"b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277\": container with ID starting with b8bfab475353b26e4d24d620ab9e40013109baa3e4ee0caae1848a093803a277 not found: ID does not exist" Sep 15 21:43:53.442119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.442037 2552 scope.go:117] "RemoveContainer" containerID="4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22" Sep 15 21:43:53.450762 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.450748 2552 scope.go:117] "RemoveContainer" containerID="4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22" Sep 15 21:43:53.451038 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:53.451019 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22\": container with ID starting with 4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22 not found: ID does not exist" containerID="4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22" Sep 15 21:43:53.451089 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.451045 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22"} err="failed to get container status \"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22\": rpc error: code = NotFound desc = could not find container \"4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22\": container with ID starting with 4dff0d8fd05e040fec5bb30782f2f5f7d81673c089789d6f68d3890c75b56e22 not found: ID does not exist" Sep 15 21:43:53.456453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.456434 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:43:53.458945 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.458927 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac4411-54cd9cff6c-zzmnz"] Sep 15 21:43:53.469717 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.469696 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:43:53.472029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.472011 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac4411-d8784c68b-44mqn"] Sep 15 21:43:53.790341 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.790314 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5" path="/var/lib/kubelet/pods/54f9d0ff-24c1-48bf-9b71-1ec0b1da9bb5/volumes" Sep 15 21:43:53.790604 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.790591 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac5ac98-79fb-474b-b43e-28983935339f" path="/var/lib/kubelet/pods/6ac5ac98-79fb-474b-b43e-28983935339f/volumes" Sep 15 21:43:53.790928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.790916 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" path="/var/lib/kubelet/pods/8846e0f9-cc3e-4c43-86ec-7eb27ef4df23/volumes" Sep 15 21:43:53.791242 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.791231 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" path="/var/lib/kubelet/pods/dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7/volumes" Sep 15 21:43:53.791867 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:53.791853 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2da6711-1939-415c-9ae6-4536dbeed30d" path="/var/lib/kubelet/pods/e2da6711-1939-415c-9ae6-4536dbeed30d/volumes" Sep 15 21:43:56.092328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.092296 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:43:56.092731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.092351 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:43:56.543318 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.543278 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" podUID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" containerName="main" containerID="cri-o://ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d" gracePeriod=45 Sep 15 21:43:56.556175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.556142 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" podUID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" containerName="main" containerID="cri-o://c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88" gracePeriod=45 Sep 15 21:43:56.881752 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.881727 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:43:56.910110 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.910082 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:43:56.979276 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979252 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979293 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979325 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979396 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979373 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979410 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979441 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979479 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979532 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979563 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqqkb\" (UniqueName: \"kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb\") pod \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\" (UID: \"91155c4a-e0dd-4bfb-81da-a6456b26a13c\") " Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979575 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home" (OuterVolumeSpecName: "home") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979609 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979646 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx67b\" (UniqueName: \"kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979671 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs\") pod \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\" (UID: \"f51657fa-fdd7-4fdd-b5e2-beec15482a55\") " Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979623 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache" (OuterVolumeSpecName: "model-cache") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979693 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home" (OuterVolumeSpecName: "home") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.979817 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.979669 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache" (OuterVolumeSpecName: "model-cache") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.980583 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.980700 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981303 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981326 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981348 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981362 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981375 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.981627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.981387 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.991166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.991139 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb" (OuterVolumeSpecName: "kube-api-access-gqqkb") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "kube-api-access-gqqkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:56.991477 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.991173 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm" (OuterVolumeSpecName: "dshm") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.991586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.991173 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:56.991586 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.991433 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm" (OuterVolumeSpecName: "dshm") pod "91155c4a-e0dd-4bfb-81da-a6456b26a13c" (UID: "91155c4a-e0dd-4bfb-81da-a6456b26a13c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.992391 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.992370 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:56.992713 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:56.992686 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b" (OuterVolumeSpecName: "kube-api-access-tx67b") pod "f51657fa-fdd7-4fdd-b5e2-beec15482a55" (UID: "f51657fa-fdd7-4fdd-b5e2-beec15482a55"). InnerVolumeSpecName "kube-api-access-tx67b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.048135 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.048109 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:43:57.082112 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082054 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082112 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082091 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082134 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082196 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082235 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082270 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082264 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082305 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082362 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.082473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082373 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hqn5\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082428 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert\") pod \"726a252e-90d8-4518-9024-d0062c2b242e\" (UID: \"726a252e-90d8-4518-9024-d0062c2b242e\") " Sep 15 21:43:57.082719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082494 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data" (OuterVolumeSpecName: "istio-data") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.082719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082510 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.082719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082618 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.082908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082805 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.082908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082844 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.082908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082862 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.082908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082870 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:57.082908 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082877 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91155c4a-e0dd-4bfb-81da-a6456b26a13c-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082919 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082936 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51657fa-fdd7-4fdd-b5e2-beec15482a55-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082951 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91155c4a-e0dd-4bfb-81da-a6456b26a13c-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082964 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqkb\" (UniqueName: \"kubernetes.io/projected/91155c4a-e0dd-4bfb-81da-a6456b26a13c-kube-api-access-gqqkb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082978 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx67b\" (UniqueName: \"kubernetes.io/projected/f51657fa-fdd7-4fdd-b5e2-beec15482a55-kube-api-access-tx67b\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.083157 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.082993 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51657fa-fdd7-4fdd-b5e2-beec15482a55-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.084377 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.084351 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token" (OuterVolumeSpecName: "istio-token") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.084653 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.084631 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:43:57.084742 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.084713 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.084793 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.084741 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5" (OuterVolumeSpecName: "kube-api-access-7hqn5") pod "726a252e-90d8-4518-9024-d0062c2b242e" (UID: "726a252e-90d8-4518-9024-d0062c2b242e"). InnerVolumeSpecName "kube-api-access-7hqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.183764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.183731 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqn5\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-kube-api-access-7hqn5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.183764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.183764 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726a252e-90d8-4518-9024-d0062c2b242e-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.183764 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.183775 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726a252e-90d8-4518-9024-d0062c2b242e-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.184172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.183783 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726a252e-90d8-4518-9024-d0062c2b242e-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.184172 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.183791 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726a252e-90d8-4518-9024-d0062c2b242e-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.463707 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.463672 2552 generic.go:356] "Generic (PLEG): container finished" podID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" containerID="ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d" exitCode=0 Sep 15 21:43:57.463904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.463720 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" event={"ID":"91155c4a-e0dd-4bfb-81da-a6456b26a13c","Type":"ContainerDied","Data":"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d"} Sep 15 21:43:57.463904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.463738 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" Sep 15 21:43:57.463904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.463763 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9" event={"ID":"91155c4a-e0dd-4bfb-81da-a6456b26a13c","Type":"ContainerDied","Data":"6fd33d24bb51479c5ed5bf4ad9a42372cc258d5360b0b9cc9f86b87140dbb1c5"} Sep 15 21:43:57.463904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.463783 2552 scope.go:117] "RemoveContainer" containerID="ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d" Sep 15 21:43:57.465728 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.465703 2552 generic.go:356] "Generic (PLEG): container finished" podID="726a252e-90d8-4518-9024-d0062c2b242e" containerID="7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7" exitCode=0 Sep 15 21:43:57.465891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.465759 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" Sep 15 21:43:57.465891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.465791 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerDied","Data":"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7"} Sep 15 21:43:57.466023 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.466003 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s" event={"ID":"726a252e-90d8-4518-9024-d0062c2b242e","Type":"ContainerDied","Data":"dfd81275467a8429ad8b42101e1f93dce6c0a756a5bbda1a566f73b35d4be33c"} Sep 15 21:43:57.467433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.467355 2552 generic.go:356] "Generic (PLEG): container finished" podID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" containerID="c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88" exitCode=0 Sep 15 21:43:57.467433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.467380 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" event={"ID":"f51657fa-fdd7-4fdd-b5e2-beec15482a55","Type":"ContainerDied","Data":"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88"} Sep 15 21:43:57.467433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.467411 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" event={"ID":"f51657fa-fdd7-4fdd-b5e2-beec15482a55","Type":"ContainerDied","Data":"7e1c5343c07a3acd3159bf013d27b44f3d8408926dc3fe41816d8362a4cdf638"} Sep 15 21:43:57.467433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.467426 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk" Sep 15 21:43:57.474043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.474026 2552 scope.go:117] "RemoveContainer" containerID="ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d" Sep 15 21:43:57.474340 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:57.474289 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d\": container with ID starting with ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d not found: ID does not exist" containerID="ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d" Sep 15 21:43:57.474340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.474323 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d"} err="failed to get container status \"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d\": rpc error: code = NotFound desc = could not find container \"ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d\": container with ID starting with ac1705424a71b0b6f67fca832b15d188df800e486250abcaf582e72d2be48f9d not found: ID does not exist" Sep 15 21:43:57.474340 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.474339 2552 scope.go:117] "RemoveContainer" containerID="7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7" Sep 15 21:43:57.483528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.483512 2552 scope.go:117] "RemoveContainer" containerID="0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd" Sep 15 21:43:57.490366 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.490344 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:43:57.491862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.491840 2552 scope.go:117] "RemoveContainer" containerID="7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7" Sep 15 21:43:57.492097 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:57.492078 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7\": container with ID starting with 7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7 not found: ID does not exist" containerID="7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7" Sep 15 21:43:57.492169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.492102 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7"} err="failed to get container status \"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7\": rpc error: code = NotFound desc = could not find container \"7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7\": container with ID starting with 7f9496f05438933b8069810f68bf8bdc9fc50970e0c6b4e93eb11f0d4e7c6ea7 not found: ID does not exist" Sep 15 21:43:57.492169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.492118 2552 scope.go:117] "RemoveContainer" containerID="0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd" Sep 15 21:43:57.492376 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:57.492356 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd\": container with ID starting with 0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd not found: ID does not exist" containerID="0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd" Sep 15 21:43:57.492431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.492379 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd"} err="failed to get container status \"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd\": rpc error: code = NotFound desc = could not find container \"0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd\": container with ID starting with 0a657ffdf9bb4d9723192e6d5a19d251ef380ea85fc286b5dfef25e350e676dd not found: ID does not exist" Sep 15 21:43:57.492431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.492392 2552 scope.go:117] "RemoveContainer" containerID="c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88" Sep 15 21:43:57.493570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.493551 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/embedding-w0-b99d7b-kserve-c5dbc44ff-lsww9"] Sep 15 21:43:57.500088 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.500075 2552 scope.go:117] "RemoveContainer" containerID="c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88" Sep 15 21:43:57.500309 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:43:57.500293 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88\": container with ID starting with c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88 not found: ID does not exist" containerID="c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88" Sep 15 21:43:57.500364 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.500314 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88"} err="failed to get container status \"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88\": rpc error: code = NotFound desc = could not find container \"c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88\": container with ID starting with c48a5499442edbb787d6911aefc485da7909dbe06018b72d1b9e7c2b6d0f5f88 not found: ID does not exist" Sep 15 21:43:57.505814 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.505794 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:43:57.517521 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.517500 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb0f0e-openshift-default-6b8f495db-t2t2s"] Sep 15 21:43:57.532102 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.532083 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:43:57.536275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.536257 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/premium-w0-b99d7b-kserve-6b47885795-dlxvk"] Sep 15 21:43:57.791513 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.791436 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726a252e-90d8-4518-9024-d0062c2b242e" path="/var/lib/kubelet/pods/726a252e-90d8-4518-9024-d0062c2b242e/volumes" Sep 15 21:43:57.791852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.791812 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" path="/var/lib/kubelet/pods/91155c4a-e0dd-4bfb-81da-a6456b26a13c/volumes" Sep 15 21:43:57.792193 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:43:57.792180 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" path="/var/lib/kubelet/pods/f51657fa-fdd7-4fdd-b5e2-beec15482a55/volumes" Sep 15 21:44:00.262180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:00.262150 2552 scope.go:117] "RemoveContainer" containerID="0981798aefcbd934ab094dfb228a7fa85090e9150b3852af0f2512fd1f31aa8e" Sep 15 21:44:00.627823 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:00.627789 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:44:00.628076 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:00.628048 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" containerID="cri-o://1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd" gracePeriod=30 Sep 15 21:44:05.880439 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.880417 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:44:05.960683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960620 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.960683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960652 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.960683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960668 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.960683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960684 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960743 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhgk4\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960764 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960780 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960871 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960897 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert\") pod \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\" (UID: \"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db\") " Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960894 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.960935 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:05.961019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961004 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:05.961328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961122 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:05.961328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961140 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:05.961328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961153 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:05.961328 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961244 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:44:05.961517 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.961362 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:05.963037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.963015 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:44:05.963154 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.963136 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4" (OuterVolumeSpecName: "kube-api-access-zhgk4") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "kube-api-access-zhgk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:05.963419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.963396 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:05.963496 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:05.963416 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" (UID: "8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:06.061971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.061944 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhgk4\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-kube-api-access-zhgk4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.061971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.061967 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.061971 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.061976 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.062164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.061985 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.062164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.061993 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.062164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.062000 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:44:06.535728 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.535695 2552 generic.go:356] "Generic (PLEG): container finished" podID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerID="1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd" exitCode=0 Sep 15 21:44:06.535918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.535745 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" Sep 15 21:44:06.535918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.535750 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerDied","Data":"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd"} Sep 15 21:44:06.535918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.535873 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv" event={"ID":"8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db","Type":"ContainerDied","Data":"d0c8c5c37ad3872f35c07d40c5e28e29820d782e1925d5888af33b3c117dbe2c"} Sep 15 21:44:06.535918 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.535898 2552 scope.go:117] "RemoveContainer" containerID="1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd" Sep 15 21:44:06.552476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.552457 2552 scope.go:117] "RemoveContainer" containerID="35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c" Sep 15 21:44:06.560618 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.560602 2552 scope.go:117] "RemoveContainer" containerID="1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd" Sep 15 21:44:06.560924 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:44:06.560905 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd\": container with ID starting with 1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd not found: ID does not exist" containerID="1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd" Sep 15 21:44:06.561009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.560936 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd"} err="failed to get container status \"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd\": rpc error: code = NotFound desc = could not find container \"1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd\": container with ID starting with 1ce73b30c4838c8f7b5f2889ebe4629bdff77b9a4b38836a7bda64834a4e6cfd not found: ID does not exist" Sep 15 21:44:06.561009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.560958 2552 scope.go:117] "RemoveContainer" containerID="35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c" Sep 15 21:44:06.561261 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:44:06.561244 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c\": container with ID starting with 35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c not found: ID does not exist" containerID="35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c" Sep 15 21:44:06.561301 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.561266 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c"} err="failed to get container status \"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c\": rpc error: code = NotFound desc = could not find container \"35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c\": container with ID starting with 35d67f4ba36aea715c6499080f22436415091cc037cdf2415dfd114e1b22007c not found: ID does not exist" Sep 15 21:44:06.561638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.561620 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:44:06.565718 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:06.565698 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ac4411-openshift-default-76ddd8dd56-qwqjv"] Sep 15 21:44:07.790221 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:44:07.790190 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" path="/var/lib/kubelet/pods/8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db/volumes" Sep 15 21:45:00.151715 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.151682 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6"] Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152056 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10ea75f-743b-43d0-bd22-7b470a796fd0" containerName="revoke-keys" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152068 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10ea75f-743b-43d0-bd22-7b470a796fd0" containerName="revoke-keys" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152079 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152085 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152095 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" containerName="main" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152101 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" containerName="main" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152109 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" containerName="payload-processing" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152114 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" containerName="payload-processing" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152121 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152127 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152134 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152139 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152147 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" containerName="payload-processing" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152152 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" containerName="payload-processing" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152162 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac5ac98-79fb-474b-b43e-28983935339f" containerName="revoke-keys" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152168 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac5ac98-79fb-474b-b43e-28983935339f" containerName="revoke-keys" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152174 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152179 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152188 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" containerName="main" Sep 15 21:45:00.152186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152194 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" containerName="main" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152200 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152206 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152213 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2da6711-1939-415c-9ae6-4536dbeed30d" containerName="payload-pre-processing" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152219 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2da6711-1939-415c-9ae6-4536dbeed30d" containerName="payload-pre-processing" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152271 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea37b66-d37c-4fb5-a5d1-e4ce3f9e1ac7" containerName="payload-processing" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152278 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152284 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8846e0f9-cc3e-4c43-86ec-7eb27ef4df23" containerName="payload-processing" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152290 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf6c929-1e0b-4798-bfb4-d88ac2f78d0b" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152295 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152300 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="91155c4a-e0dd-4bfb-81da-a6456b26a13c" containerName="main" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152306 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10ea75f-743b-43d0-bd22-7b470a796fd0" containerName="revoke-keys" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152311 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152316 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf73f6e-ba2b-4a0e-8b1f-8c21db8444db" containerName="istio-proxy" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152322 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2da6711-1939-415c-9ae6-4536dbeed30d" containerName="payload-pre-processing" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152326 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51657fa-fdd7-4fdd-b5e2-beec15482a55" containerName="main" Sep 15 21:45:00.152862 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.152331 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac5ac98-79fb-474b-b43e-28983935339f" containerName="revoke-keys" Sep 15 21:45:00.155456 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.155439 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" Sep 15 21:45:00.159166 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.159141 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2"] Sep 15 21:45:00.159465 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.159453 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.159507 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.159467 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.159551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.159542 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="726a252e-90d8-4518-9024-d0062c2b242e" containerName="istio-proxy" Sep 15 21:45:00.162420 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.162405 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" Sep 15 21:45:00.163009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.162994 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:45:00.165001 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.164983 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" Sep 15 21:45:00.174719 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.174701 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" Sep 15 21:45:00.174890 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.174868 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6"] Sep 15 21:45:00.178170 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.178147 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2"] Sep 15 21:45:00.452115 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.452034 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6"] Sep 15 21:45:00.457253 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:45:00.457229 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0f1349_d754_473f_82c3_a9c0b8edca24.slice/crio-01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb WatchSource:0}: Error finding container 01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb: Status 404 returned error can't find the container with id 01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb Sep 15 21:45:00.467950 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.467673 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2"] Sep 15 21:45:00.474867 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:45:00.474839 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a12954_691b_40c7_af23_ab05a599f196.slice/crio-f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102 WatchSource:0}: Error finding container f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102: Status 404 returned error can't find the container with id f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102 Sep 15 21:45:00.839278 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.839248 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" event={"ID":"55a12954-691b-40c7-af23-ab05a599f196","Type":"ContainerStarted","Data":"d5c753c8cc8f8fbfafa963359eb99aff05941e0719480741b8a19bea762b6db1"} Sep 15 21:45:00.839417 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.839299 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" event={"ID":"55a12954-691b-40c7-af23-ab05a599f196","Type":"ContainerStarted","Data":"f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102"} Sep 15 21:45:00.840928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.840890 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" event={"ID":"eb0f1349-d754-473f-82c3-a9c0b8edca24","Type":"ContainerStarted","Data":"68254c5092b750ccf17432a3a691ed84d712b090377d7e4bd6912694cc9768a2"} Sep 15 21:45:00.841029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.840934 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" event={"ID":"eb0f1349-d754-473f-82c3-a9c0b8edca24","Type":"ContainerStarted","Data":"01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb"} Sep 15 21:45:00.858408 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.858369 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" podStartSLOduration=0.858354923 podStartE2EDuration="858.354923ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:00.856425473 +0000 UTC m=+1741.491577587" watchObservedRunningTime="2026-09-15 21:45:00.858354923 +0000 UTC m=+1741.493507034" Sep 15 21:45:00.874250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:00.874212 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" podStartSLOduration=0.874200096 podStartE2EDuration="874.200096ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:00.871018933 +0000 UTC m=+1741.506171044" watchObservedRunningTime="2026-09-15 21:45:00.874200096 +0000 UTC m=+1741.509352207" Sep 15 21:45:01.847915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:01.847810 2552 generic.go:356] "Generic (PLEG): container finished" podID="eb0f1349-d754-473f-82c3-a9c0b8edca24" containerID="68254c5092b750ccf17432a3a691ed84d712b090377d7e4bd6912694cc9768a2" exitCode=0 Sep 15 21:45:01.847915 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:01.847862 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" event={"ID":"eb0f1349-d754-473f-82c3-a9c0b8edca24","Type":"ContainerDied","Data":"68254c5092b750ccf17432a3a691ed84d712b090377d7e4bd6912694cc9768a2"} Sep 15 21:45:01.849501 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:01.849479 2552 generic.go:356] "Generic (PLEG): container finished" podID="55a12954-691b-40c7-af23-ab05a599f196" containerID="d5c753c8cc8f8fbfafa963359eb99aff05941e0719480741b8a19bea762b6db1" exitCode=0 Sep 15 21:45:01.849618 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:01.849525 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" event={"ID":"55a12954-691b-40c7-af23-ab05a599f196","Type":"ContainerDied","Data":"d5c753c8cc8f8fbfafa963359eb99aff05941e0719480741b8a19bea762b6db1"} Sep 15 21:45:02.997091 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:02.997067 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" Sep 15 21:45:03.003684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.003664 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" Sep 15 21:45:03.862045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.862021 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" Sep 15 21:45:03.862158 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.862068 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2" event={"ID":"55a12954-691b-40c7-af23-ab05a599f196","Type":"ContainerDied","Data":"f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102"} Sep 15 21:45:03.862158 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.862116 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9386b15f613d4074f20f6ebc2b7770ea9fe23bae4059d640995e9c4cc28f102" Sep 15 21:45:03.863552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.863496 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" event={"ID":"eb0f1349-d754-473f-82c3-a9c0b8edca24","Type":"ContainerDied","Data":"01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb"} Sep 15 21:45:03.863552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.863516 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6" Sep 15 21:45:03.863552 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:03.863535 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b86c10988d879a1e20508a4c929a36fa86f59c7cf09f390526781b2e6d70cb" Sep 15 21:45:59.942176 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:59.942152 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:45:59.951300 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:45:59.951280 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:46:00.339474 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:00.339401 2552 scope.go:117] "RemoveContainer" containerID="a1fdcc6ddb59a304d0018029c0589259a9a8d32cbce4c8349dcf2ab765cf2c10" Sep 15 21:46:56.462261 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462225 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462729 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a12954-691b-40c7-af23-ab05a599f196" containerName="cleanup" Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462744 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a12954-691b-40c7-af23-ab05a599f196" containerName="cleanup" Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462760 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0f1349-d754-473f-82c3-a9c0b8edca24" containerName="cleanup" Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462766 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0f1349-d754-473f-82c3-a9c0b8edca24" containerName="cleanup" Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462849 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a12954-691b-40c7-af23-ab05a599f196" containerName="cleanup" Sep 15 21:46:56.464570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.462860 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0f1349-d754-473f-82c3-a9c0b8edca24" containerName="cleanup" Sep 15 21:46:56.465675 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.465656 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.468632 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.468614 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b2c5fe-openshift-default-dockercfg-d4q7l\"" Sep 15 21:46:56.487707 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.487684 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:46:56.596415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596379 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596420 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596454 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596498 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596560 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgjh\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596597 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596625 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596906 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596685 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.596906 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.596711 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.697977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698012 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698044 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698086 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698122 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgjh\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698155 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698186 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698247 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698275 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698272 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698447 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698702 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698673 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698759 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698733 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.698816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.698764 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.699109 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.699088 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.700456 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.700433 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.700654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.700638 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.706139 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.706115 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.706332 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.706314 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgjh\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh\") pod \"e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:56.775906 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:56.775781 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:57.058891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.058863 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:46:57.061128 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:46:57.061100 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86469e65_f10e_424b_b5aa_e80e7499a30b.slice/crio-86f8ee15f24f0280da95fc151788077f5bc0fafea5515ca9be9af1a0c53822aa WatchSource:0}: Error finding container 86f8ee15f24f0280da95fc151788077f5bc0fafea5515ca9be9af1a0c53822aa: Status 404 returned error can't find the container with id 86f8ee15f24f0280da95fc151788077f5bc0fafea5515ca9be9af1a0c53822aa Sep 15 21:46:57.063279 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.063250 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:46:57.063379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.063316 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:46:57.063379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.063346 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:46:57.538772 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.538737 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerStarted","Data":"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1"} Sep 15 21:46:57.539189 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.538800 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerStarted","Data":"86f8ee15f24f0280da95fc151788077f5bc0fafea5515ca9be9af1a0c53822aa"} Sep 15 21:46:57.560167 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.560119 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podStartSLOduration=1.560104145 podStartE2EDuration="1.560104145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:57.557739004 +0000 UTC m=+1858.192891115" watchObservedRunningTime="2026-09-15 21:46:57.560104145 +0000 UTC m=+1858.195256255" Sep 15 21:46:57.776201 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:57.776164 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:58.781499 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:58.781474 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:59.553466 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:59.553426 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:46:59.557151 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:46:59.557123 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:47:15.292169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.292121 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:47500->10.133.0.100:15021: read: connection reset by peer" start-of-body= Sep 15 21:47:15.292608 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.292311 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:47500->10.133.0.100:15021: read: connection reset by peer" Sep 15 21:47:15.648320 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.648292 2552 generic.go:356] "Generic (PLEG): container finished" podID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerID="390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1" exitCode=0 Sep 15 21:47:15.648490 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.648371 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerDied","Data":"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1"} Sep 15 21:47:15.648774 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.648759 2552 scope.go:117] "RemoveContainer" containerID="390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1" Sep 15 21:47:15.649795 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.649768 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:47:15.649873 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.649860 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:47:15.649916 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:15.649898 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:47:16.660652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:16.660611 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerStarted","Data":"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba"} Sep 15 21:47:16.776467 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:16.776426 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:47:16.776674 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:16.776649 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Sep 15 21:47:16.776722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:16.776707 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Sep 15 21:47:17.776808 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:17.776770 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Sep 15 21:47:17.777186 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:17.776823 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Sep 15 21:47:18.776975 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:18.776936 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Sep 15 21:47:18.777433 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:18.776991 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Sep 15 21:47:19.777036 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:19.777000 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Sep 15 21:47:19.777399 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:19.777053 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Sep 15 21:47:20.776241 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:20.776200 2552 patch_prober.go:28] interesting pod/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Sep 15 21:47:20.776414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:20.776261 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Sep 15 21:47:21.780213 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:21.780184 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:47:21.780655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:21.780641 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:47:21.781333 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:21.781311 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:47:36.572316 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.572237 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:47:36.575251 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.575234 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.577398 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.577374 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0ba80\"" Sep 15 21:47:36.577523 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.577418 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0ba80-dockercfg-d26gw\"" Sep 15 21:47:36.587860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.587813 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:47:36.609019 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.608991 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:47:36.611579 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.611562 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.624137 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.624113 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:47:36.655881 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.655856 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swstt\" (UniqueName: \"kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.656000 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.655908 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.757126 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.757086 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.757295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.757156 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.757295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.757186 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kwq\" (UniqueName: \"kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.757295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.757211 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swstt\" (UniqueName: \"kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.757802 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.757781 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.765488 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.765457 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swstt\" (UniqueName: \"kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt\") pod \"payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.858444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.858358 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.858444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.858431 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kwq\" (UniqueName: \"kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.859334 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.859307 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.866081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.866059 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kwq\" (UniqueName: \"kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq\") pod \"payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:36.886001 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.885966 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:36.921976 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:36.921939 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:37.165331 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.165307 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:47:37.167424 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:47:37.167396 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978d5b56_8a49_4a8c_b600_4ad1afdc93d3.slice/crio-e381130fe5e63fdee4a8511e48d57f948a9cb8e60d57eecac8c8fedf6ccdb5af WatchSource:0}: Error finding container e381130fe5e63fdee4a8511e48d57f948a9cb8e60d57eecac8c8fedf6ccdb5af: Status 404 returned error can't find the container with id e381130fe5e63fdee4a8511e48d57f948a9cb8e60d57eecac8c8fedf6ccdb5af Sep 15 21:47:37.205283 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.205259 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:47:37.206061 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:47:37.206038 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134e7905_2fb2_4698_96dc_caf656e9a207.slice/crio-c576fec902a40b6f24733f1bfe97cdca26b960a877e62fce92c33a91ad67226b WatchSource:0}: Error finding container c576fec902a40b6f24733f1bfe97cdca26b960a877e62fce92c33a91ad67226b: Status 404 returned error can't find the container with id c576fec902a40b6f24733f1bfe97cdca26b960a877e62fce92c33a91ad67226b Sep 15 21:47:37.783638 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.783605 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" event={"ID":"978d5b56-8a49-4a8c-b600-4ad1afdc93d3","Type":"ContainerStarted","Data":"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f"} Sep 15 21:47:37.784140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.783699 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" event={"ID":"978d5b56-8a49-4a8c-b600-4ad1afdc93d3","Type":"ContainerStarted","Data":"e381130fe5e63fdee4a8511e48d57f948a9cb8e60d57eecac8c8fedf6ccdb5af"} Sep 15 21:47:37.784140 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.783736 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:37.790413 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.790391 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:47:37.790542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.790436 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" event={"ID":"134e7905-2fb2-4698-96dc-caf656e9a207","Type":"ContainerStarted","Data":"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520"} Sep 15 21:47:37.790542 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.790458 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" event={"ID":"134e7905-2fb2-4698-96dc-caf656e9a207","Type":"ContainerStarted","Data":"c576fec902a40b6f24733f1bfe97cdca26b960a877e62fce92c33a91ad67226b"} Sep 15 21:47:37.802232 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.802189 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" podStartSLOduration=1.802173288 podStartE2EDuration="1.802173288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:37.799217204 +0000 UTC m=+1898.434369328" watchObservedRunningTime="2026-09-15 21:47:37.802173288 +0000 UTC m=+1898.437325400" Sep 15 21:47:37.815059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:37.815018 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" podStartSLOduration=1.8150043500000002 podStartE2EDuration="1.81500435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:37.814038 +0000 UTC m=+1898.449190126" watchObservedRunningTime="2026-09-15 21:47:37.81500435 +0000 UTC m=+1898.450156461" Sep 15 21:47:48.793607 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:48.793576 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:47:48.794009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:47:48.793644 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:48:08.189563 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:48:08.189522 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:48:08.302414 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:48:08.302375 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb_134e7905-2fb2-4698-96dc-caf656e9a207/payload-processing/0.log" Sep 15 21:48:54.429427 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:48:54.429341 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:48:54.540595 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:48:54.540566 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb_134e7905-2fb2-4698-96dc-caf656e9a207/payload-processing/0.log" Sep 15 21:49:00.378183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:00.378147 2552 scope.go:117] "RemoveContainer" containerID="204895fe0e6be0f717dc9f282a09ef8095360cb5cb0cbdc8eb47d40d02b69fba" Sep 15 21:49:00.387009 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:00.386984 2552 scope.go:117] "RemoveContainer" containerID="42c16a60eddc12b898f2caf9e01341fcde22c0b0bea53d73f4e9e838e3b56b4c" Sep 15 21:49:02.758122 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.758089 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:02.763269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.763249 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.766464 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.766439 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5dc07b\"" Sep 15 21:49:02.766573 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.766544 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5dc07b-dockercfg-bsxnt\"" Sep 15 21:49:02.771199 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.770949 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsbs\" (UniqueName: \"kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.771199 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.771120 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.773039 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.773016 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:02.872255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.872218 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.872458 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.872262 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsbs\" (UniqueName: \"kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.872956 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.872932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:02.880605 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:02.880582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsbs\" (UniqueName: \"kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs\") pod \"payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:03.074692 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:03.074622 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:03.366368 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:03.366333 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:03.368552 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:49:03.368524 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85870704_67d7_46c5_8250_878f69c14093.slice/crio-f6faacfcbe68bef1917fe707715c81fb58c0e18f5f91a603e43526426739c005 WatchSource:0}: Error finding container f6faacfcbe68bef1917fe707715c81fb58c0e18f5f91a603e43526426739c005: Status 404 returned error can't find the container with id f6faacfcbe68bef1917fe707715c81fb58c0e18f5f91a603e43526426739c005 Sep 15 21:49:04.288008 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:04.287972 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" event={"ID":"85870704-67d7-46c5-8250-878f69c14093","Type":"ContainerStarted","Data":"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004"} Sep 15 21:49:04.288370 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:04.288040 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" event={"ID":"85870704-67d7-46c5-8250-878f69c14093","Type":"ContainerStarted","Data":"f6faacfcbe68bef1917fe707715c81fb58c0e18f5f91a603e43526426739c005"} Sep 15 21:49:04.288370 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:04.288135 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:04.307244 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:04.307195 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" podStartSLOduration=2.307181508 podStartE2EDuration="2.307181508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:49:04.306091307 +0000 UTC m=+1984.941243419" watchObservedRunningTime="2026-09-15 21:49:04.307181508 +0000 UTC m=+1984.942333617" Sep 15 21:49:15.297419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:15.297386 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:28.717269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.717234 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp"] Sep 15 21:49:28.721003 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.720986 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:28.724132 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.724112 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:49:28.733904 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.733879 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp"] Sep 15 21:49:28.790749 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.790719 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp\" (UID: \"8c32d949-5d54-4274-9cb3-6f08643fe6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:28.891453 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.891418 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp\" (UID: \"8c32d949-5d54-4274-9cb3-6f08643fe6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:28.892044 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:28.892026 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp\" (UID: \"8c32d949-5d54-4274-9cb3-6f08643fe6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:29.031113 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:29.031039 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:29.313109 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:29.313083 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp"] Sep 15 21:49:29.315551 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:49:29.315525 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c32d949_5d54_4274_9cb3_6f08643fe6bd.slice/crio-eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c WatchSource:0}: Error finding container eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c: Status 404 returned error can't find the container with id eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c Sep 15 21:49:29.434476 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:29.434435 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" event={"ID":"8c32d949-5d54-4274-9cb3-6f08643fe6bd","Type":"ContainerStarted","Data":"eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c"} Sep 15 21:49:30.442698 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:30.442666 2552 generic.go:356] "Generic (PLEG): container finished" podID="8c32d949-5d54-4274-9cb3-6f08643fe6bd" containerID="ff9f503d022859a91ae449216234c653c3e6620294c5d1c4ff9602377f760fd3" exitCode=0 Sep 15 21:49:30.443164 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:30.442755 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" event={"ID":"8c32d949-5d54-4274-9cb3-6f08643fe6bd","Type":"ContainerDied","Data":"ff9f503d022859a91ae449216234c653c3e6620294c5d1c4ff9602377f760fd3"} Sep 15 21:49:31.588320 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:31.588296 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:31.720046 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:31.719962 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca\") pod \"8c32d949-5d54-4274-9cb3-6f08643fe6bd\" (UID: \"8c32d949-5d54-4274-9cb3-6f08643fe6bd\") " Sep 15 21:49:31.720350 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:31.720328 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c32d949-5d54-4274-9cb3-6f08643fe6bd" (UID: "8c32d949-5d54-4274-9cb3-6f08643fe6bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:49:31.821299 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:31.821269 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c32d949-5d54-4274-9cb3-6f08643fe6bd-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:49:32.455412 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:32.455385 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" Sep 15 21:49:32.455591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:32.455412 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp" event={"ID":"8c32d949-5d54-4274-9cb3-6f08643fe6bd","Type":"ContainerDied","Data":"eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c"} Sep 15 21:49:32.455591 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:32.455455 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb77122e27c779c7d9701a31e0cdf2cc7ca95136587152b0272ea4a3c8add02c" Sep 15 21:49:39.920085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:39.920050 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:39.920495 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:39.920286 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" podUID="85870704-67d7-46c5-8250-878f69c14093" containerName="payload-pre-processing" containerID="cri-o://03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004" gracePeriod=30 Sep 15 21:49:40.121329 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.121282 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp"] Sep 15 21:49:40.123997 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.123976 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5dc07b-4ljxp"] Sep 15 21:49:40.168177 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.168158 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:40.302003 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.301929 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdsbs\" (UniqueName: \"kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs\") pod \"85870704-67d7-46c5-8250-878f69c14093\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " Sep 15 21:49:40.302155 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.302011 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume\") pod \"85870704-67d7-46c5-8250-878f69c14093\" (UID: \"85870704-67d7-46c5-8250-878f69c14093\") " Sep 15 21:49:40.302406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.302379 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85870704-67d7-46c5-8250-878f69c14093" (UID: "85870704-67d7-46c5-8250-878f69c14093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:49:40.304037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.304017 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs" (OuterVolumeSpecName: "kube-api-access-qdsbs") pod "85870704-67d7-46c5-8250-878f69c14093" (UID: "85870704-67d7-46c5-8250-878f69c14093"). InnerVolumeSpecName "kube-api-access-qdsbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:49:40.403284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.403254 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85870704-67d7-46c5-8250-878f69c14093-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:49:40.403284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.403280 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsbs\" (UniqueName: \"kubernetes.io/projected/85870704-67d7-46c5-8250-878f69c14093-kube-api-access-qdsbs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:49:40.503750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.503718 2552 generic.go:356] "Generic (PLEG): container finished" podID="85870704-67d7-46c5-8250-878f69c14093" containerID="03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004" exitCode=0 Sep 15 21:49:40.503899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.503763 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" Sep 15 21:49:40.503899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.503798 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" event={"ID":"85870704-67d7-46c5-8250-878f69c14093","Type":"ContainerDied","Data":"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004"} Sep 15 21:49:40.503899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.503869 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp" event={"ID":"85870704-67d7-46c5-8250-878f69c14093","Type":"ContainerDied","Data":"f6faacfcbe68bef1917fe707715c81fb58c0e18f5f91a603e43526426739c005"} Sep 15 21:49:40.503899 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.503893 2552 scope.go:117] "RemoveContainer" containerID="03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004" Sep 15 21:49:40.513000 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.512986 2552 scope.go:117] "RemoveContainer" containerID="03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004" Sep 15 21:49:40.513271 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:49:40.513252 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004\": container with ID starting with 03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004 not found: ID does not exist" containerID="03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004" Sep 15 21:49:40.513317 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.513279 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004"} err="failed to get container status \"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004\": rpc error: code = NotFound desc = could not find container \"03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004\": container with ID starting with 03e4c91760cac294fef5b7ae0484a76b2ff79a3bcce37cd80a34fab58af51004 not found: ID does not exist" Sep 15 21:49:40.529661 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.529636 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:40.536787 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:40.536767 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5dc07b-8688cd68bb-l7zfp"] Sep 15 21:49:41.791345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:41.791298 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85870704-67d7-46c5-8250-878f69c14093" path="/var/lib/kubelet/pods/85870704-67d7-46c5-8250-878f69c14093/volumes" Sep 15 21:49:41.792029 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:41.791810 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c32d949-5d54-4274-9cb3-6f08643fe6bd" path="/var/lib/kubelet/pods/8c32d949-5d54-4274-9cb3-6f08643fe6bd/volumes" Sep 15 21:49:50.407215 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407180 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn"] Sep 15 21:49:50.407603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407550 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c32d949-5d54-4274-9cb3-6f08643fe6bd" containerName="revoke-keys" Sep 15 21:49:50.407603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407561 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c32d949-5d54-4274-9cb3-6f08643fe6bd" containerName="revoke-keys" Sep 15 21:49:50.407603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407577 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85870704-67d7-46c5-8250-878f69c14093" containerName="payload-pre-processing" Sep 15 21:49:50.407603 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407583 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="85870704-67d7-46c5-8250-878f69c14093" containerName="payload-pre-processing" Sep 15 21:49:50.407757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407640 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="85870704-67d7-46c5-8250-878f69c14093" containerName="payload-pre-processing" Sep 15 21:49:50.407757 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.407648 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c32d949-5d54-4274-9cb3-6f08643fe6bd" containerName="revoke-keys" Sep 15 21:49:50.411786 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.411769 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:50.414106 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.414083 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:49:50.419394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.419366 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn"] Sep 15 21:49:50.493601 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.493576 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn\" (UID: \"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:50.594330 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.594302 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn\" (UID: \"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:50.594944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.594917 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn\" (UID: \"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:50.724445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.724358 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:50.998760 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:50.998732 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn"] Sep 15 21:49:51.000354 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:49:51.000328 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df3cc0e_cf7b_4c4c_a630_2ef70ea6d0e7.slice/crio-fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76 WatchSource:0}: Error finding container fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76: Status 404 returned error can't find the container with id fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76 Sep 15 21:49:51.572952 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:51.572915 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" event={"ID":"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7","Type":"ContainerStarted","Data":"a48860f74b5fdd7265d45fac552b5b9861966fc6a6ab87fae732ae2a1a228968"} Sep 15 21:49:51.573305 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:51.572966 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" event={"ID":"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7","Type":"ContainerStarted","Data":"fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76"} Sep 15 21:49:51.590174 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:51.590125 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" podStartSLOduration=1.5901077209999999 podStartE2EDuration="1.590107721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:49:51.587854799 +0000 UTC m=+2032.223006915" watchObservedRunningTime="2026-09-15 21:49:51.590107721 +0000 UTC m=+2032.225259880" Sep 15 21:49:52.580520 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:52.580482 2552 generic.go:356] "Generic (PLEG): container finished" podID="1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" containerID="a48860f74b5fdd7265d45fac552b5b9861966fc6a6ab87fae732ae2a1a228968" exitCode=0 Sep 15 21:49:52.580906 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:52.580572 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" event={"ID":"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7","Type":"ContainerDied","Data":"a48860f74b5fdd7265d45fac552b5b9861966fc6a6ab87fae732ae2a1a228968"} Sep 15 21:49:53.727450 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:53.727422 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:49:53.821306 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:53.821278 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca\") pod \"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7\" (UID: \"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7\") " Sep 15 21:49:53.821705 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:53.821659 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" (UID: "1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:49:53.922546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:53.922522 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:49:54.600745 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:54.600705 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" event={"ID":"1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7","Type":"ContainerDied","Data":"fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76"} Sep 15 21:49:54.600931 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:54.600756 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe69a9fa4ac4e3d13993666e60465b35fd34c139152aff9f722bae4e12f9ae76" Sep 15 21:49:54.600931 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:49:54.600720 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn" Sep 15 21:50:00.412551 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:00.412518 2552 scope.go:117] "RemoveContainer" containerID="0896df76ed0278ac50abff8890f95d64a56b958df05995b84d5f7d5559b707bf" Sep 15 21:50:00.421310 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:00.421286 2552 scope.go:117] "RemoveContainer" containerID="a1ba5f6fb3096ee1766d74a055a6d5c0a82c00cc9d46c2488a1e6a9552202f48" Sep 15 21:50:00.429208 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:00.429187 2552 scope.go:117] "RemoveContainer" containerID="ad83929d784ebe3805704f8be73b499d2001c30325bf91c9303d66dbb2c5ae14" Sep 15 21:50:06.435442 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:06.435400 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn"] Sep 15 21:50:06.440150 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:06.440118 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b2c5fe-csvfn"] Sep 15 21:50:07.790609 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:07.790577 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" path="/var/lib/kubelet/pods/1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7/volumes" Sep 15 21:50:55.530526 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.530490 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:50:55.531035 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.530754 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" containerID="cri-o://28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba" gracePeriod=30 Sep 15 21:50:55.919473 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.919444 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp"] Sep 15 21:50:55.919807 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.919794 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" containerName="revoke-keys" Sep 15 21:50:55.919879 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.919808 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" containerName="revoke-keys" Sep 15 21:50:55.919921 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.919890 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df3cc0e-cf7b-4c4c-a630-2ef70ea6d0e7" containerName="revoke-keys" Sep 15 21:50:55.922962 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.922943 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:55.927614 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.927594 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:50:55.936614 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:55.936588 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp"] Sep 15 21:50:56.044469 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.044443 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp\" (UID: \"71632883-8682-4b9b-93cc-0d1dc07df653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:56.145013 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.144982 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp\" (UID: \"71632883-8682-4b9b-93cc-0d1dc07df653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:56.145567 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.145547 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp\" (UID: \"71632883-8682-4b9b-93cc-0d1dc07df653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:56.233388 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.233315 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:56.505180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.505156 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp"] Sep 15 21:50:56.506488 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:50:56.506466 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71632883_8682_4b9b_93cc_0d1dc07df653.slice/crio-c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee WatchSource:0}: Error finding container c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee: Status 404 returned error can't find the container with id c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee Sep 15 21:50:56.955429 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.955400 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" event={"ID":"71632883-8682-4b9b-93cc-0d1dc07df653","Type":"ContainerStarted","Data":"c805a131dfb3c44ddf117b651abda17cf61b5734200eb1c8b6379728acb51d48"} Sep 15 21:50:56.955738 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.955449 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" event={"ID":"71632883-8682-4b9b-93cc-0d1dc07df653","Type":"ContainerStarted","Data":"c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee"} Sep 15 21:50:56.972903 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:56.972855 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" podStartSLOduration=1.972840888 podStartE2EDuration="1.972840888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:56.971615019 +0000 UTC m=+2097.606767131" watchObservedRunningTime="2026-09-15 21:50:56.972840888 +0000 UTC m=+2097.607992989" Sep 15 21:50:57.962618 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:57.962580 2552 generic.go:356] "Generic (PLEG): container finished" podID="71632883-8682-4b9b-93cc-0d1dc07df653" containerID="c805a131dfb3c44ddf117b651abda17cf61b5734200eb1c8b6379728acb51d48" exitCode=0 Sep 15 21:50:57.962618 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:57.962616 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" event={"ID":"71632883-8682-4b9b-93cc-0d1dc07df653","Type":"ContainerDied","Data":"c805a131dfb3c44ddf117b651abda17cf61b5734200eb1c8b6379728acb51d48"} Sep 15 21:50:59.104231 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.104208 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:59.177948 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.177917 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca\") pod \"71632883-8682-4b9b-93cc-0d1dc07df653\" (UID: \"71632883-8682-4b9b-93cc-0d1dc07df653\") " Sep 15 21:50:59.178224 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.178205 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71632883-8682-4b9b-93cc-0d1dc07df653" (UID: "71632883-8682-4b9b-93cc-0d1dc07df653"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:50:59.279119 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.279057 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71632883-8682-4b9b-93cc-0d1dc07df653-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:50:59.975466 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.975429 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" event={"ID":"71632883-8682-4b9b-93cc-0d1dc07df653","Type":"ContainerDied","Data":"c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee"} Sep 15 21:50:59.975639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.975492 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1aed7deb9197c7c8a1f2b5ef4a6298895521a361b316f5d62c04e5b5f11ddee" Sep 15 21:50:59.975639 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.975448 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp" Sep 15 21:50:59.982181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.982154 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:50:59.997245 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:50:59.997226 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:51:00.778863 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.778821 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:51:00.893022 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.892993 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893037 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgjh\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893072 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893104 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893122 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893175 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893149 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893180 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893207 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893269 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data" (OuterVolumeSpecName: "istio-data") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:00.893441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893280 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket\") pod \"86469e65-f10e-424b-b5aa-e80e7499a30b\" (UID: \"86469e65-f10e-424b-b5aa-e80e7499a30b\") " Sep 15 21:51:00.893441 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893410 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:00.893712 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893446 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:00.893712 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893495 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:00.893712 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893635 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:00.893712 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893688 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.893895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893721 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.893895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893737 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.893895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.893751 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.895257 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.895235 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh" (OuterVolumeSpecName: "kube-api-access-szgjh") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "kube-api-access-szgjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:00.895415 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.895395 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:51:00.895626 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.895601 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:00.895720 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.895622 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token" (OuterVolumeSpecName: "istio-token") pod "86469e65-f10e-424b-b5aa-e80e7499a30b" (UID: "86469e65-f10e-424b-b5aa-e80e7499a30b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:00.982751 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.982723 2552 generic.go:356] "Generic (PLEG): container finished" podID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerID="28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba" exitCode=0 Sep 15 21:51:00.982891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.982767 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" Sep 15 21:51:00.982891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.982806 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerDied","Data":"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba"} Sep 15 21:51:00.982891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.982876 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl" event={"ID":"86469e65-f10e-424b-b5aa-e80e7499a30b","Type":"ContainerDied","Data":"86f8ee15f24f0280da95fc151788077f5bc0fafea5515ca9be9af1a0c53822aa"} Sep 15 21:51:00.983014 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.982899 2552 scope.go:117] "RemoveContainer" containerID="28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba" Sep 15 21:51:00.992566 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.992551 2552 scope.go:117] "RemoveContainer" containerID="390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1" Sep 15 21:51:00.994419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.994395 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgjh\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-kube-api-access-szgjh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.994508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.994424 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.994508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.994438 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86469e65-f10e-424b-b5aa-e80e7499a30b-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.994508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.994451 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:00.994508 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:00.994464 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86469e65-f10e-424b-b5aa-e80e7499a30b-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:01.000583 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.000564 2552 scope.go:117] "RemoveContainer" containerID="28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba" Sep 15 21:51:01.000821 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:01.000802 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba\": container with ID starting with 28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba not found: ID does not exist" containerID="28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba" Sep 15 21:51:01.000886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.000853 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba"} err="failed to get container status \"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba\": rpc error: code = NotFound desc = could not find container \"28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba\": container with ID starting with 28a26a7be1241ccc3dd06de81b33211f394fb8637f92ec1b96fd3d2b6b9755ba not found: ID does not exist" Sep 15 21:51:01.000886 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.000872 2552 scope.go:117] "RemoveContainer" containerID="390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1" Sep 15 21:51:01.001129 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:01.001111 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1\": container with ID starting with 390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1 not found: ID does not exist" containerID="390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1" Sep 15 21:51:01.001184 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.001135 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1"} err="failed to get container status \"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1\": rpc error: code = NotFound desc = could not find container \"390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1\": container with ID starting with 390ef6fce38be405e8c5ebf90a4ed87a6858de16d4fdc706b54e9eeb4c3219b1 not found: ID does not exist" Sep 15 21:51:01.004905 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.004884 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:51:01.009112 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.009090 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2c5fe-openshift-default-7885bfc56d-5z4bl"] Sep 15 21:51:01.791169 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:01.791137 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" path="/var/lib/kubelet/pods/86469e65-f10e-424b-b5aa-e80e7499a30b/volumes" Sep 15 21:51:06.563570 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.563537 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:51:06.564052 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.563781 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" podUID="134e7905-2fb2-4698-96dc-caf656e9a207" containerName="payload-processing" containerID="cri-o://ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520" gracePeriod=30 Sep 15 21:51:06.573059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.573033 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:51:06.573292 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.573251 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" podUID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" containerName="payload-pre-processing" containerID="cri-o://a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f" gracePeriod=30 Sep 15 21:51:06.770946 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.770891 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp"] Sep 15 21:51:06.778141 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.778113 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0ba80-czwtp"] Sep 15 21:51:06.851783 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.851758 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:51:06.854321 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.854304 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:51:06.946250 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946222 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume\") pod \"134e7905-2fb2-4698-96dc-caf656e9a207\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " Sep 15 21:51:06.946423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946259 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42kwq\" (UniqueName: \"kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq\") pod \"134e7905-2fb2-4698-96dc-caf656e9a207\" (UID: \"134e7905-2fb2-4698-96dc-caf656e9a207\") " Sep 15 21:51:06.946423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946287 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swstt\" (UniqueName: \"kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt\") pod \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " Sep 15 21:51:06.946423 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946353 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume\") pod \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\" (UID: \"978d5b56-8a49-4a8c-b600-4ad1afdc93d3\") " Sep 15 21:51:06.946688 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946661 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "134e7905-2fb2-4698-96dc-caf656e9a207" (UID: "134e7905-2fb2-4698-96dc-caf656e9a207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:06.946805 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.946700 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "978d5b56-8a49-4a8c-b600-4ad1afdc93d3" (UID: "978d5b56-8a49-4a8c-b600-4ad1afdc93d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:06.948339 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.948316 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt" (OuterVolumeSpecName: "kube-api-access-swstt") pod "978d5b56-8a49-4a8c-b600-4ad1afdc93d3" (UID: "978d5b56-8a49-4a8c-b600-4ad1afdc93d3"). InnerVolumeSpecName "kube-api-access-swstt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:06.948406 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:06.948357 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq" (OuterVolumeSpecName: "kube-api-access-42kwq") pod "134e7905-2fb2-4698-96dc-caf656e9a207" (UID: "134e7905-2fb2-4698-96dc-caf656e9a207"). InnerVolumeSpecName "kube-api-access-42kwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:07.019548 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.019520 2552 generic.go:356] "Generic (PLEG): container finished" podID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" containerID="a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f" exitCode=0 Sep 15 21:51:07.019693 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.019574 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" Sep 15 21:51:07.019693 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.019604 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" event={"ID":"978d5b56-8a49-4a8c-b600-4ad1afdc93d3","Type":"ContainerDied","Data":"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f"} Sep 15 21:51:07.019693 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.019665 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9" event={"ID":"978d5b56-8a49-4a8c-b600-4ad1afdc93d3","Type":"ContainerDied","Data":"e381130fe5e63fdee4a8511e48d57f948a9cb8e60d57eecac8c8fedf6ccdb5af"} Sep 15 21:51:07.019693 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.019692 2552 scope.go:117] "RemoveContainer" containerID="a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f" Sep 15 21:51:07.020960 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.020937 2552 generic.go:356] "Generic (PLEG): container finished" podID="134e7905-2fb2-4698-96dc-caf656e9a207" containerID="ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520" exitCode=0 Sep 15 21:51:07.021074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.020964 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" event={"ID":"134e7905-2fb2-4698-96dc-caf656e9a207","Type":"ContainerDied","Data":"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520"} Sep 15 21:51:07.021074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.020982 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" Sep 15 21:51:07.021074 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.021013 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb" event={"ID":"134e7905-2fb2-4698-96dc-caf656e9a207","Type":"ContainerDied","Data":"c576fec902a40b6f24733f1bfe97cdca26b960a877e62fce92c33a91ad67226b"} Sep 15 21:51:07.030290 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.030271 2552 scope.go:117] "RemoveContainer" containerID="a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f" Sep 15 21:51:07.030596 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:07.030576 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f\": container with ID starting with a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f not found: ID does not exist" containerID="a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f" Sep 15 21:51:07.030645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.030604 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f"} err="failed to get container status \"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f\": rpc error: code = NotFound desc = could not find container \"a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f\": container with ID starting with a5a8b230d70e40fcd6ce8eac277bae6c876bcc902d5d1f3dbb83ed75d95a4c8f not found: ID does not exist" Sep 15 21:51:07.030645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.030619 2552 scope.go:117] "RemoveContainer" containerID="ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520" Sep 15 21:51:07.038797 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.038779 2552 scope.go:117] "RemoveContainer" containerID="ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520" Sep 15 21:51:07.039083 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:07.039062 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520\": container with ID starting with ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520 not found: ID does not exist" containerID="ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520" Sep 15 21:51:07.039148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.039095 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520"} err="failed to get container status \"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520\": rpc error: code = NotFound desc = could not find container \"ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520\": container with ID starting with ab6b40da8c032c30dafa0a357c197cfc730091a588c4652dc61047fc8f2d0520 not found: ID does not exist" Sep 15 21:51:07.046312 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.046292 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:51:07.047816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.047798 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134e7905-2fb2-4698-96dc-caf656e9a207-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:07.047891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.047818 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42kwq\" (UniqueName: \"kubernetes.io/projected/134e7905-2fb2-4698-96dc-caf656e9a207-kube-api-access-42kwq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:07.047891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.047842 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swstt\" (UniqueName: \"kubernetes.io/projected/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-kube-api-access-swstt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:07.047891 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.047859 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978d5b56-8a49-4a8c-b600-4ad1afdc93d3-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:07.059846 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.055614 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0ba80-6b5457bcf7-g8vbb"] Sep 15 21:51:07.070055 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.070030 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:51:07.073038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.072991 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0ba80-84b8454c9d-xkbx9"] Sep 15 21:51:07.789877 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.789815 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="134e7905-2fb2-4698-96dc-caf656e9a207" path="/var/lib/kubelet/pods/134e7905-2fb2-4698-96dc-caf656e9a207/volumes" Sep 15 21:51:07.790234 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.790192 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71632883-8682-4b9b-93cc-0d1dc07df653" path="/var/lib/kubelet/pods/71632883-8682-4b9b-93cc-0d1dc07df653/volumes" Sep 15 21:51:07.790521 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:07.790507 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" path="/var/lib/kubelet/pods/978d5b56-8a49-4a8c-b600-4ad1afdc93d3/volumes" Sep 15 21:51:13.782461 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782429 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7"] Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782797 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782810 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782818 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71632883-8682-4b9b-93cc-0d1dc07df653" containerName="revoke-keys" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782823 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="71632883-8682-4b9b-93cc-0d1dc07df653" containerName="revoke-keys" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782848 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" containerName="payload-pre-processing" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782857 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" containerName="payload-pre-processing" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782876 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134e7905-2fb2-4698-96dc-caf656e9a207" containerName="payload-processing" Sep 15 21:51:13.782929 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782885 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="134e7905-2fb2-4698-96dc-caf656e9a207" containerName="payload-processing" Sep 15 21:51:13.783181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782958 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:13.783181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782966 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="71632883-8682-4b9b-93cc-0d1dc07df653" containerName="revoke-keys" Sep 15 21:51:13.783181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782974 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="134e7905-2fb2-4698-96dc-caf656e9a207" containerName="payload-processing" Sep 15 21:51:13.783181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782979 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:13.783181 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.782984 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="978d5b56-8a49-4a8c-b600-4ad1afdc93d3" containerName="payload-pre-processing" Sep 15 21:51:13.787376 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.787353 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:13.791983 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.791966 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:51:13.800085 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.800063 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7"] Sep 15 21:51:13.901037 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:13.901009 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7\" (UID: \"950f220a-1a9d-41f9-952a-2dafe33f2ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:14.002061 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:14.002034 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7\" (UID: \"950f220a-1a9d-41f9-952a-2dafe33f2ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:14.002650 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:14.002628 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7\" (UID: \"950f220a-1a9d-41f9-952a-2dafe33f2ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:14.098295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:14.098226 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:14.375047 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:14.375018 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7"] Sep 15 21:51:14.378457 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:51:14.378421 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950f220a_1a9d_41f9_952a_2dafe33f2ac5.slice/crio-972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9 WatchSource:0}: Error finding container 972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9: Status 404 returned error can't find the container with id 972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9 Sep 15 21:51:15.068790 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:15.068752 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" event={"ID":"950f220a-1a9d-41f9-952a-2dafe33f2ac5","Type":"ContainerStarted","Data":"eef0d17ed44f1304afefe02a2698da944ae65b10ae2ee47ff2aa761a9864b839"} Sep 15 21:51:15.069168 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:15.068813 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" event={"ID":"950f220a-1a9d-41f9-952a-2dafe33f2ac5","Type":"ContainerStarted","Data":"972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9"} Sep 15 21:51:16.075595 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.075558 2552 generic.go:356] "Generic (PLEG): container finished" podID="950f220a-1a9d-41f9-952a-2dafe33f2ac5" containerID="eef0d17ed44f1304afefe02a2698da944ae65b10ae2ee47ff2aa761a9864b839" exitCode=0 Sep 15 21:51:16.076033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.075633 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" event={"ID":"950f220a-1a9d-41f9-952a-2dafe33f2ac5","Type":"ContainerDied","Data":"eef0d17ed44f1304afefe02a2698da944ae65b10ae2ee47ff2aa761a9864b839"} Sep 15 21:51:16.203206 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.203184 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:16.323694 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.323662 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca\") pod \"950f220a-1a9d-41f9-952a-2dafe33f2ac5\" (UID: \"950f220a-1a9d-41f9-952a-2dafe33f2ac5\") " Sep 15 21:51:16.324033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.324011 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "950f220a-1a9d-41f9-952a-2dafe33f2ac5" (UID: "950f220a-1a9d-41f9-952a-2dafe33f2ac5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:16.424647 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:16.424621 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f220a-1a9d-41f9-952a-2dafe33f2ac5-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:17.082738 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:17.082713 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" Sep 15 21:51:17.083148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:17.082709 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7" event={"ID":"950f220a-1a9d-41f9-952a-2dafe33f2ac5","Type":"ContainerDied","Data":"972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9"} Sep 15 21:51:17.083148 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:17.082860 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="972a146b0fef3573fb6834d43c1f36010b3c03aae80416115bb725f422e772d9" Sep 15 21:51:24.490038 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.489993 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2"] Sep 15 21:51:24.492876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.492848 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-59e243-29825145-w9sv2"] Sep 15 21:51:24.523746 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.523717 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:51:24.523992 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.523964 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" podUID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" containerName="payload-processing" containerID="cri-o://231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69" gracePeriod=30 Sep 15 21:51:24.723033 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.723003 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7"] Sep 15 21:51:24.731608 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.731579 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59e243-np2q7"] Sep 15 21:51:24.777872 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.777850 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:51:24.894722 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.894687 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume\") pod \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " Sep 15 21:51:24.894910 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.894804 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq59p\" (UniqueName: \"kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p\") pod \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\" (UID: \"1deb4f5e-7a96-4156-9fa1-50b34415ab77\") " Sep 15 21:51:24.895135 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.895115 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1deb4f5e-7a96-4156-9fa1-50b34415ab77" (UID: "1deb4f5e-7a96-4156-9fa1-50b34415ab77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:24.896964 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.896944 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p" (OuterVolumeSpecName: "kube-api-access-mq59p") pod "1deb4f5e-7a96-4156-9fa1-50b34415ab77" (UID: "1deb4f5e-7a96-4156-9fa1-50b34415ab77"). InnerVolumeSpecName "kube-api-access-mq59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:24.995797 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.995768 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1deb4f5e-7a96-4156-9fa1-50b34415ab77-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:24.995797 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:24.995790 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq59p\" (UniqueName: \"kubernetes.io/projected/1deb4f5e-7a96-4156-9fa1-50b34415ab77-kube-api-access-mq59p\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:25.129672 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.129637 2552 generic.go:356] "Generic (PLEG): container finished" podID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" containerID="231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69" exitCode=0 Sep 15 21:51:25.129895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.129684 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" Sep 15 21:51:25.129895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.129693 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" event={"ID":"1deb4f5e-7a96-4156-9fa1-50b34415ab77","Type":"ContainerDied","Data":"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69"} Sep 15 21:51:25.129895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.129753 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h" event={"ID":"1deb4f5e-7a96-4156-9fa1-50b34415ab77","Type":"ContainerDied","Data":"68815204f8a756dd18df0f3d3ed2ebef44279f5b06d6fe2a1bb7fc07212a32e1"} Sep 15 21:51:25.129895 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.129781 2552 scope.go:117] "RemoveContainer" containerID="231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69" Sep 15 21:51:25.139034 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.139019 2552 scope.go:117] "RemoveContainer" containerID="231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69" Sep 15 21:51:25.139312 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:25.139294 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69\": container with ID starting with 231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69 not found: ID does not exist" containerID="231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69" Sep 15 21:51:25.139359 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.139320 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69"} err="failed to get container status \"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69\": rpc error: code = NotFound desc = could not find container \"231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69\": container with ID starting with 231345cc45eca05e82b7bd4b0294936793fd8753f44a24784fda4ecf44feba69 not found: ID does not exist" Sep 15 21:51:25.157788 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.157759 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:51:25.160100 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.160082 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59e243-7b5c848b75-6z22h"] Sep 15 21:51:25.793648 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.793603 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" path="/var/lib/kubelet/pods/1deb4f5e-7a96-4156-9fa1-50b34415ab77/volumes" Sep 15 21:51:25.794145 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.794030 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a12954-691b-40c7-af23-ab05a599f196" path="/var/lib/kubelet/pods/55a12954-691b-40c7-af23-ab05a599f196/volumes" Sep 15 21:51:25.794369 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:25.794350 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950f220a-1a9d-41f9-952a-2dafe33f2ac5" path="/var/lib/kubelet/pods/950f220a-1a9d-41f9-952a-2dafe33f2ac5/volumes" Sep 15 21:51:31.408086 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.408052 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:51:31.408462 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.408299 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" containerID="cri-o://5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa" gracePeriod=30 Sep 15 21:51:31.771180 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771103 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6"] Sep 15 21:51:31.771444 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771431 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:31.771485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771446 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="86469e65-f10e-424b-b5aa-e80e7499a30b" containerName="istio-proxy" Sep 15 21:51:31.771485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771460 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950f220a-1a9d-41f9-952a-2dafe33f2ac5" containerName="revoke-keys" Sep 15 21:51:31.771485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771465 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="950f220a-1a9d-41f9-952a-2dafe33f2ac5" containerName="revoke-keys" Sep 15 21:51:31.771485 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771482 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" containerName="payload-processing" Sep 15 21:51:31.771604 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771487 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" containerName="payload-processing" Sep 15 21:51:31.771604 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771564 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="950f220a-1a9d-41f9-952a-2dafe33f2ac5" containerName="revoke-keys" Sep 15 21:51:31.771604 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.771572 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1deb4f5e-7a96-4156-9fa1-50b34415ab77" containerName="payload-processing" Sep 15 21:51:31.776097 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.776078 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:31.778079 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.778063 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:51:31.782934 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.782911 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6"] Sep 15 21:51:31.952192 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:31.952163 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6\" (UID: \"c93c7bc8-a006-4fe3-8767-87e96b330e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:32.052651 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:32.052576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6\" (UID: \"c93c7bc8-a006-4fe3-8767-87e96b330e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:32.053173 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:32.053154 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6\" (UID: \"c93c7bc8-a006-4fe3-8767-87e96b330e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:32.088108 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:32.088084 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:32.365024 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:32.364989 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6"] Sep 15 21:51:32.366165 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:51:32.366143 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93c7bc8_a006_4fe3_8767_87e96b330e64.slice/crio-13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642 WatchSource:0}: Error finding container 13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642: Status 404 returned error can't find the container with id 13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642 Sep 15 21:51:33.177059 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:33.177026 2552 generic.go:356] "Generic (PLEG): container finished" podID="c93c7bc8-a006-4fe3-8767-87e96b330e64" containerID="8b362c680bab7bc06ea7bb9123f92bfe56573d8b6e8226e302cb5f967e19a4d5" exitCode=0 Sep 15 21:51:33.177445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:33.177112 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" event={"ID":"c93c7bc8-a006-4fe3-8767-87e96b330e64","Type":"ContainerDied","Data":"8b362c680bab7bc06ea7bb9123f92bfe56573d8b6e8226e302cb5f967e19a4d5"} Sep 15 21:51:33.177445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:33.177169 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" event={"ID":"c93c7bc8-a006-4fe3-8767-87e96b330e64","Type":"ContainerStarted","Data":"13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642"} Sep 15 21:51:34.330920 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:34.330900 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:34.474346 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:34.474273 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca\") pod \"c93c7bc8-a006-4fe3-8767-87e96b330e64\" (UID: \"c93c7bc8-a006-4fe3-8767-87e96b330e64\") " Sep 15 21:51:34.474598 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:34.474578 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c93c7bc8-a006-4fe3-8767-87e96b330e64" (UID: "c93c7bc8-a006-4fe3-8767-87e96b330e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:34.575240 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:34.575214 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c7bc8-a006-4fe3-8767-87e96b330e64-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:35.190182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:35.190155 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" Sep 15 21:51:35.190182 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:35.190173 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6" event={"ID":"c93c7bc8-a006-4fe3-8767-87e96b330e64","Type":"ContainerDied","Data":"13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642"} Sep 15 21:51:35.190394 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:35.190211 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ad8b830154599e2bdd65a1baf5c37cb23aefce62e76a4676e7f9d0f3268642" Sep 15 21:51:36.653928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.653901 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:51:36.793376 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793311 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793376 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793365 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793395 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793413 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793433 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq4zz\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793459 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793546 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793522 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793547 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793571 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket\") pod \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\" (UID: \"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5\") " Sep 15 21:51:36.793852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793691 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:36.793852 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793764 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:36.794081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793821 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:36.794081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793864 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.794081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793882 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.794081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793934 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:36.794081 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.793943 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:36.795545 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.795525 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:36.795682 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.795665 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:36.795944 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.795924 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz" (OuterVolumeSpecName: "kube-api-access-rq4zz") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "kube-api-access-rq4zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:36.796045 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.795967 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" (UID: "7b89ff44-b881-4ad3-8fc9-f52026cd8bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:51:36.894535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894509 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894535 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894533 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894547 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4zz\" (UniqueName: \"kubernetes.io/projected/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-kube-api-access-rq4zz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894556 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894564 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894572 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:36.894725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:36.894579 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:37.206167 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.206134 2552 generic.go:356] "Generic (PLEG): container finished" podID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerID="5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa" exitCode=0 Sep 15 21:51:37.206327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.206177 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerDied","Data":"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa"} Sep 15 21:51:37.206327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.206189 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" Sep 15 21:51:37.206327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.206241 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g" event={"ID":"7b89ff44-b881-4ad3-8fc9-f52026cd8bd5","Type":"ContainerDied","Data":"4636507e16d24a32c796e00405e459c6cb324657454bf0cf57f10abc1a556811"} Sep 15 21:51:37.206327 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.206262 2552 scope.go:117] "RemoveContainer" containerID="5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa" Sep 15 21:51:37.215854 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.215817 2552 scope.go:117] "RemoveContainer" containerID="018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139" Sep 15 21:51:37.223973 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.223954 2552 scope.go:117] "RemoveContainer" containerID="5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa" Sep 15 21:51:37.224216 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:37.224195 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa\": container with ID starting with 5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa not found: ID does not exist" containerID="5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa" Sep 15 21:51:37.224284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.224228 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa"} err="failed to get container status \"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa\": rpc error: code = NotFound desc = could not find container \"5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa\": container with ID starting with 5f77508642de73b9b5e60020f908bfadcb1fd05a7185c6c70ef337080e5c6dfa not found: ID does not exist" Sep 15 21:51:37.224284 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.224251 2552 scope.go:117] "RemoveContainer" containerID="018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139" Sep 15 21:51:37.224514 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:37.224496 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139\": container with ID starting with 018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139 not found: ID does not exist" containerID="018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139" Sep 15 21:51:37.224555 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.224519 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139"} err="failed to get container status \"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139\": rpc error: code = NotFound desc = could not find container \"018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139\": container with ID starting with 018fb22500834bd167164b91e39d1e50186fa2faa50ec01530475f3e6c5f7139 not found: ID does not exist" Sep 15 21:51:37.231624 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.231605 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:51:37.236749 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.236721 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59e243-openshift-default-594bc5c896-lmd5g"] Sep 15 21:51:37.789571 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:37.789541 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" path="/var/lib/kubelet/pods/7b89ff44-b881-4ad3-8fc9-f52026cd8bd5/volumes" Sep 15 21:51:42.354922 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:42.354885 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6"] Sep 15 21:51:42.361376 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:42.358800 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-38a3a3-29825145-hvdr6"] Sep 15 21:51:42.683050 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:42.682995 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6"] Sep 15 21:51:42.686814 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:42.686786 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-38a3a3-42hf6"] Sep 15 21:51:43.789725 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:43.789683 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93c7bc8-a006-4fe3-8767-87e96b330e64" path="/var/lib/kubelet/pods/c93c7bc8-a006-4fe3-8767-87e96b330e64/volumes" Sep 15 21:51:43.790104 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:43.790001 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0f1349-d754-473f-82c3-a9c0b8edca24" path="/var/lib/kubelet/pods/eb0f1349-d754-473f-82c3-a9c0b8edca24/volumes" Sep 15 21:51:49.878198 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:49.878164 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:51:49.878737 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:49.878386 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" containerID="cri-o://d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a" gracePeriod=30 Sep 15 21:51:55.124998 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.124976 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:51:55.238627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238556 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238627 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238600 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238631 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctzzt\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238677 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238692 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238717 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238745 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238783 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.238855 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.238805 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data\") pod \"3482fb2c-5c54-49e2-9861-fb3992fed332\" (UID: \"3482fb2c-5c54-49e2-9861-fb3992fed332\") " Sep 15 21:51:55.239269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.239027 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:55.239269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.239105 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:55.239269 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.239249 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:55.239431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.239311 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:55.239431 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.239260 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data" (OuterVolumeSpecName: "istio-data") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:55.240860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.240808 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:51:55.240860 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.240852 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt" (OuterVolumeSpecName: "kube-api-access-ctzzt") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "kube-api-access-ctzzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:55.240996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.240896 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:51:55.240996 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.240930 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token" (OuterVolumeSpecName: "istio-token") pod "3482fb2c-5c54-49e2-9861-fb3992fed332" (UID: "3482fb2c-5c54-49e2-9861-fb3992fed332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:55.313547 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.313519 2552 generic.go:356] "Generic (PLEG): container finished" podID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerID="d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a" exitCode=0 Sep 15 21:51:55.313654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.313572 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" Sep 15 21:51:55.313654 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.313607 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerDied","Data":"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a"} Sep 15 21:51:55.313736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.313661 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9" event={"ID":"3482fb2c-5c54-49e2-9861-fb3992fed332","Type":"ContainerDied","Data":"5a4f390d8bc6403344c0fa052b7d782202ad5b56ea793869b4a3138ddd49e782"} Sep 15 21:51:55.313736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.313683 2552 scope.go:117] "RemoveContainer" containerID="d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a" Sep 15 21:51:55.323273 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.323254 2552 scope.go:117] "RemoveContainer" containerID="a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c" Sep 15 21:51:55.331139 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.331122 2552 scope.go:117] "RemoveContainer" containerID="d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a" Sep 15 21:51:55.331410 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:55.331390 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a\": container with ID starting with d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a not found: ID does not exist" containerID="d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a" Sep 15 21:51:55.331456 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.331418 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a"} err="failed to get container status \"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a\": rpc error: code = NotFound desc = could not find container \"d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a\": container with ID starting with d2160bfe3b4477aa7d4cb2d0f956ee6f453b8b9258c95c74cf57cd2058c8ec8a not found: ID does not exist" Sep 15 21:51:55.331456 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.331435 2552 scope.go:117] "RemoveContainer" containerID="a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c" Sep 15 21:51:55.331695 ip-10-0-133-61 kubenswrapper[2552]: E0915 21:51:55.331677 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c\": container with ID starting with a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c not found: ID does not exist" containerID="a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c" Sep 15 21:51:55.331734 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.331702 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c"} err="failed to get container status \"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c\": rpc error: code = NotFound desc = could not find container \"a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c\": container with ID starting with a504c44c3f7f87bdc76dbc26c975cf1489c00b32996cad136a4050f2b415b53c not found: ID does not exist" Sep 15 21:51:55.336276 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.336256 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:51:55.340313 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340293 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340315 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340329 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzzt\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-kube-api-access-ctzzt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340344 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340358 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482fb2c-5c54-49e2-9861-fb3992fed332-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340371 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340384 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340397 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.340419 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.340408 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482fb2c-5c54-49e2-9861-fb3992fed332-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Sep 15 21:51:55.341566 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.341548 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-38a3a3-openshift-default-6cbb76bc64-4fvj9"] Sep 15 21:51:55.796078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:51:55.796041 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" path="/var/lib/kubelet/pods/3482fb2c-5c54-49e2-9861-fb3992fed332/volumes" Sep 15 21:52:00.474454 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:52:00.474427 2552 scope.go:117] "RemoveContainer" containerID="d5c753c8cc8f8fbfafa963359eb99aff05941e0719480741b8a19bea762b6db1" Sep 15 21:52:00.482528 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:52:00.482509 2552 scope.go:117] "RemoveContainer" containerID="68254c5092b750ccf17432a3a691ed84d712b090377d7e4bd6912694cc9768a2" Sep 15 21:56:00.021811 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:56:00.021783 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:56:00.035208 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:56:00.035185 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:56:00.555335 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:56:00.555306 2552 scope.go:117] "RemoveContainer" containerID="ff9f503d022859a91ae449216234c653c3e6620294c5d1c4ff9602377f760fd3" Sep 15 21:56:00.563655 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:56:00.563635 2552 scope.go:117] "RemoveContainer" containerID="a48860f74b5fdd7265d45fac552b5b9861966fc6a6ab87fae732ae2a1a228968" Sep 15 21:57:00.586973 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:57:00.586938 2552 scope.go:117] "RemoveContainer" containerID="c805a131dfb3c44ddf117b651abda17cf61b5734200eb1c8b6379728acb51d48" Sep 15 21:58:00.606972 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:00.606942 2552 scope.go:117] "RemoveContainer" containerID="8b362c680bab7bc06ea7bb9123f92bfe56573d8b6e8226e302cb5f967e19a4d5" Sep 15 21:58:00.615272 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:00.615252 2552 scope.go:117] "RemoveContainer" containerID="eef0d17ed44f1304afefe02a2698da944ae65b10ae2ee47ff2aa761a9864b839" Sep 15 21:58:05.554556 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:05.554526 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m9jtb_9bfeed15-c9e3-43da-bb00-8f6e0554d67e/manager/0.log" Sep 15 21:58:06.413637 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:06.413607 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-ltcrm_1f75cdf0-2576-47b7-a6bc-978931526051/manager/0.log" Sep 15 21:58:06.432531 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:06.432507 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-ltcrm_1f75cdf0-2576-47b7-a6bc-978931526051/copy-manifests/0.log" Sep 15 21:58:07.399706 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:07.399678 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-72dr8_84ec2d4f-f965-48ac-a26f-9897a17c71e8/postgres/0.log" Sep 15 21:58:08.312614 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:08.312585 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-s9hnr_fb65d59f-23b3-43d3-9634-c5b9193c45a3/manager/0.log" Sep 15 21:58:08.977816 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:08.977787 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vkc2v_220af762-58bc-4141-8776-57ebfd41fd65/manager/0.log" Sep 15 21:58:09.439068 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:09.439042 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5s4kc_53015761-220d-4d64-8b2f-d317680d9ffa/discovery/0.log" Sep 15 21:58:09.555769 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:09.555738 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-4mrh4_476f847b-6bd2-4b39-82cf-9cba3b64c97e/kube-auth-proxy/0.log" Sep 15 21:58:10.071384 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:10.071353 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:58:10.203940 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:10.203910 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cdd487d68-4972n_2dedd29e-1f9c-4549-8697-44de0145c8e6/router/0.log" Sep 15 21:58:10.684731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:10.684704 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-trtsd_67585424-e2c1-471a-b126-823e0fd354bc/main/0.log" Sep 15 21:58:10.931405 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:10.931375 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tlc7h_73ecfb64-b4e3-409f-a7d9-b725d9a0c793/main/0.log" Sep 15 21:58:20.740445 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:20.740417 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6w6wz_0487501f-1be4-4d51-8ba4-8314af778987/global-pull-secret-syncer/0.log" Sep 15 21:58:21.221645 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:21.221618 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vf44d_6f3178ee-7869-410c-9852-308038c9bf62/konnectivity-agent/0.log" Sep 15 21:58:21.373911 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:21.373882 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-61.ec2.internal_f9572ef115be1ab8679034b1a50b59a2/haproxy/0.log" Sep 15 21:58:25.083242 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:25.083215 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-s9hnr_fb65d59f-23b3-43d3-9634-c5b9193c45a3/manager/0.log" Sep 15 21:58:25.590212 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:25.590183 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vkc2v_220af762-58bc-4141-8776-57ebfd41fd65/manager/0.log" Sep 15 21:58:27.713613 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:27.713586 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-8c4js_637e59d0-4000-4180-a0ca-92c166ac1061/cluster-monitoring-operator/0.log" Sep 15 21:58:28.072727 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:28.072700 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrqq_a868f3da-5875-494a-9f77-78e52ac70823/node-exporter/0.log" Sep 15 21:58:28.142600 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:28.142576 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrqq_a868f3da-5875-494a-9f77-78e52ac70823/kube-rbac-proxy/0.log" Sep 15 21:58:28.183870 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:28.183846 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrqq_a868f3da-5875-494a-9f77-78e52ac70823/init-textfile/0.log" Sep 15 21:58:30.585671 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.585637 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm"] Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586007 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586020 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586036 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586041 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586058 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586063 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586071 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93c7bc8-a006-4fe3-8767-87e96b330e64" containerName="revoke-keys" Sep 15 21:58:30.586072 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586076 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93c7bc8-a006-4fe3-8767-87e96b330e64" containerName="revoke-keys" Sep 15 21:58:30.586323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586125 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93c7bc8-a006-4fe3-8767-87e96b330e64" containerName="revoke-keys" Sep 15 21:58:30.586323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586132 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" Sep 15 21:58:30.586323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586138 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b89ff44-b881-4ad3-8fc9-f52026cd8bd5" containerName="istio-proxy" Sep 15 21:58:30.586323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586145 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.586323 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.586150 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482fb2c-5c54-49e2-9861-fb3992fed332" containerName="istio-proxy" Sep 15 21:58:30.589387 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.589371 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.591963 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.591938 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"kube-root-ca.crt\"" Sep 15 21:58:30.592100 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.591977 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"openshift-service-ca.crt\"" Sep 15 21:58:30.595079 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.595055 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7p98\"/\"default-dockercfg-87jck\"" Sep 15 21:58:30.625893 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.625870 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm"] Sep 15 21:58:30.730767 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.730745 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-sys\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.730928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.730776 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-lib-modules\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.730928 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.730813 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-podres\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.731006 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.730941 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ln6\" (UniqueName: \"kubernetes.io/projected/abde109a-5dd7-40a1-8984-42e70f647b76-kube-api-access-52ln6\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.731006 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.730971 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-proc\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832043 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832019 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-podres\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832073 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ln6\" (UniqueName: \"kubernetes.io/projected/abde109a-5dd7-40a1-8984-42e70f647b76-kube-api-access-52ln6\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832092 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-proc\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-sys\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832159 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832132 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-lib-modules\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832185 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-podres\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832209 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-proc\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832232 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-sys\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.832295 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.832240 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abde109a-5dd7-40a1-8984-42e70f647b76-lib-modules\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.858144 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.858090 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ln6\" (UniqueName: \"kubernetes.io/projected/abde109a-5dd7-40a1-8984-42e70f647b76-kube-api-access-52ln6\") pod \"perf-node-gather-daemonset-vg4rm\" (UID: \"abde109a-5dd7-40a1-8984-42e70f647b76\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:30.899150 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:30.899130 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:31.181892 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:31.181869 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm"] Sep 15 21:58:31.183976 ip-10-0-133-61 kubenswrapper[2552]: W0915 21:58:31.183950 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podabde109a_5dd7_40a1_8984_42e70f647b76.slice/crio-48f98cec0f253099ea4d895d53c962bb6dc2a8b10ee3ea54eaaeec145e346e4b WatchSource:0}: Error finding container 48f98cec0f253099ea4d895d53c962bb6dc2a8b10ee3ea54eaaeec145e346e4b: Status 404 returned error can't find the container with id 48f98cec0f253099ea4d895d53c962bb6dc2a8b10ee3ea54eaaeec145e346e4b Sep 15 21:58:31.403615 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:31.403574 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" event={"ID":"abde109a-5dd7-40a1-8984-42e70f647b76","Type":"ContainerStarted","Data":"0eeed8782def9e2fe666164c8d2f8dc68f49ad07b0a1234504a21bfc91f677a8"} Sep 15 21:58:31.403750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:31.403624 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" event={"ID":"abde109a-5dd7-40a1-8984-42e70f647b76","Type":"ContainerStarted","Data":"48f98cec0f253099ea4d895d53c962bb6dc2a8b10ee3ea54eaaeec145e346e4b"} Sep 15 21:58:31.403750 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:31.403706 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:31.434683 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:31.434606 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" podStartSLOduration=1.434593856 podStartE2EDuration="1.434593856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:31.433464241 +0000 UTC m=+2552.068616352" watchObservedRunningTime="2026-09-15 21:58:31.434593856 +0000 UTC m=+2552.069745968" Sep 15 21:58:32.089735 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:32.089707 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/2.log" Sep 15 21:58:32.093379 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:32.093356 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2ss7v_c716d930-781b-428b-b942-0129be7d46fe/console-operator/3.log" Sep 15 21:58:32.768684 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:32.768652 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d947dbf67-qlzjl_6daa122c-af09-4312-812e-44da8b4fd9e0/console/0.log" Sep 15 21:58:32.820208 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:32.820182 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-c59gt_cd41948e-b0c0-496d-83b3-4d2d492e3e86/download-server/0.log" Sep 15 21:58:33.496845 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:33.496797 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-nnvbn_e6618063-1fe3-46a2-a8b2-16e049b31415/volume-data-source-validator/0.log" Sep 15 21:58:34.713190 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:34.713157 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vwjjx_9238defc-83a1-40db-9a48-7002cd3a4010/dns/0.log" Sep 15 21:58:34.742736 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:34.742708 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vwjjx_9238defc-83a1-40db-9a48-7002cd3a4010/kube-rbac-proxy/0.log" Sep 15 21:58:34.823673 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:34.823647 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fx9kz_757e5f7e-9388-48af-bc43-6ed61d49a316/dns-node-resolver/0.log" Sep 15 21:58:35.531000 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:35.530973 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hj8x9_fa5d8f7e-3092-496c-a773-1d0ef3c3295f/node-ca/0.log" Sep 15 21:58:37.008982 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:37.008954 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5s4kc_53015761-220d-4d64-8b2f-d317680d9ffa/discovery/0.log" Sep 15 21:58:37.033078 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:37.033056 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-4mrh4_476f847b-6bd2-4b39-82cf-9cba3b64c97e/kube-auth-proxy/0.log" Sep 15 21:58:37.176133 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:37.176103 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:58:37.211633 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:37.211605 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cdd487d68-4972n_2dedd29e-1f9c-4549-8697-44de0145c8e6/router/0.log" Sep 15 21:58:37.418635 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:37.418607 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-vg4rm" Sep 15 21:58:38.109345 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.109315 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fjdlx_c71e5357-0147-4721-a31c-09e2e67ce367/serve-healthcheck-canary/0.log" Sep 15 21:58:38.712752 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.712719 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-b24hp_bd64bcee-baf1-4cf4-93fd-4c0a041dd634/insights-operator/0.log" Sep 15 21:58:38.714492 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.714472 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-b24hp_bd64bcee-baf1-4cf4-93fd-4c0a041dd634/insights-operator/1.log" Sep 15 21:58:38.889922 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.889898 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwdvd_1168da50-436b-4731-ba66-99dfe04d4b83/kube-rbac-proxy/0.log" Sep 15 21:58:38.921333 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.921312 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwdvd_1168da50-436b-4731-ba66-99dfe04d4b83/exporter/0.log" Sep 15 21:58:38.950183 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:38.950164 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwdvd_1168da50-436b-4731-ba66-99dfe04d4b83/extractor/0.log" Sep 15 21:58:41.750671 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:41.750642 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m9jtb_9bfeed15-c9e3-43da-bb00-8f6e0554d67e/manager/0.log" Sep 15 21:58:42.013676 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.013603 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5s4kc_53015761-220d-4d64-8b2f-d317680d9ffa/discovery/0.log" Sep 15 21:58:42.031946 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.031922 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-4mrh4_476f847b-6bd2-4b39-82cf-9cba3b64c97e/kube-auth-proxy/0.log" Sep 15 21:58:42.180878 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.180847 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:58:42.194823 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.194800 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cdd487d68-4972n_2dedd29e-1f9c-4549-8697-44de0145c8e6/router/0.log" Sep 15 21:58:42.595255 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.595224 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-ltcrm_1f75cdf0-2576-47b7-a6bc-978931526051/manager/0.log" Sep 15 21:58:42.616966 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:42.616940 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-ltcrm_1f75cdf0-2576-47b7-a6bc-978931526051/copy-manifests/0.log" Sep 15 21:58:44.136731 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:44.136699 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fr2sd_fb4888a5-1b7d-42e5-b85f-9a8a257007b2/openshift-lws-operator/0.log" Sep 15 21:58:44.256652 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:44.256623 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5s4kc_53015761-220d-4d64-8b2f-d317680d9ffa/discovery/0.log" Sep 15 21:58:44.281876 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:44.281848 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-4mrh4_476f847b-6bd2-4b39-82cf-9cba3b64c97e/kube-auth-proxy/0.log" Sep 15 21:58:44.378237 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:44.378205 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cjf7f_bff5fb03-322b-405e-8f20-e4767ba57873/payload-processing/0.log" Sep 15 21:58:44.394882 ip-10-0-133-61 kubenswrapper[2552]: I0915 21:58:44.394814 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cdd487d68-4972n_2dedd29e-1f9c-4549-8697-44de0145c8e6/router/0.log"